DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statements (IDS) submitted on November 29, 2024 are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statements have been considered by the examiner.
Claim Objections
Claim 1 is objected to because of the following informalities: “determines that first SSVEPs” should read “determines that the first SSVEPs” method” (line 22). Appropriate correction is required.
Claims 2-11 are objected to because of the following informalities: “A method” should read “The method” (line 1). Appropriate correction is required.
Claim 11 is objected to because of the following informality: “If the further condition is met” should read “if the further condition is met” (line 6). Appropriate correction is required.
Claim 13 is objected to because of the following informality: “the vision comprises” should read “the vision system comprises” (line 6). Appropriate correction is required.
Claim 14 is objected to because of the following informalities: “A controller” should read “The controller” (line 1). Appropriate correction is required.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitations use a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitations are: “EEG control module”, “camera control module”, “image processing module”, “display control module”, “EEG processing module”, and “electric wheelchair module” in claim 12.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 2, 6-7, 9-12, and 14 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 2 recites “preferably a convolutions neural network” in line 2. The phrase “preferably" renders the claim indefinite because it is unclear whether the limitation(s) following the phrase are part of the claimed invention. See MPEP § 2173.05(d).
Claim 6 recites “preferably flickering targets or checkerboards” in lines 2-3. The phrase “preferably" renders the claim indefinite because it is unclear whether the limitation(s) following the phrase are part of the claimed invention. See MPEP § 2173.05(d).
Claim 7 recites “viewing a part of the scene” in line 2. It is unclear if this is meant to be the same part of the scene described in claim 1, or if this is a different part of the scene. Further clarification is required.
Claim 9 recites the limitation "the screen" in line 5. There is insufficient antecedent basis for this limitation in the claim.
Claim 10 recites “preferably by means of executing steps (b1)-(b4)” in line 4. The phrase “preferably" renders the claim indefinite because it is unclear whether the limitation(s) following the phrase are part of the claimed invention. See MPEP § 2173.05(d).
Claim 11 recite “detecting an object” in line 2. It is unclear if this is meant to be the same as the object described in claim 1, or if this is a different object. Further clarification is required.
Claim 11 recites “preferably a plethora of control visual stimuli” in line 7. The phrase “preferably" renders the claim indefinite because it is unclear whether the limitation(s) following the phrase are part of the claimed invention. See MPEP § 2173.05(d).
Claim 12 recites limitations previously recited in claim 1, upon which claim 12 depends. As written, it is unclear if these elements are meant to be the same as the mobility system of claim 1, or if these are different elements, as they are preceded by “a” instead of “the” or “said”. Further clarification is required.
Claim 14 recites “preferably a convolutional neural network” in line 2. The phrase “preferably" renders the claim indefinite because it is unclear whether the limitation(s) following the phrase are part of the claimed invention. See MPEP § 2173.05(d).
The following is a quotation of 35 U.S.C. 112(d):
(d) REFERENCE IN DEPENDENT FORMS.—Subject to subsection (e), a claim in dependent form shall contain a reference to a claim previously set forth and then specify a further limitation of the subject matter claimed. A claim in dependent form shall be construed to incorporate by reference all the limitations of the claim to which it refers.
The following is a quotation of pre-AIA 35 U.S.C. 112, fourth paragraph:
Subject to the following paragraph [i.e., the fifth paragraph of pre-AIA 35 U.S.C. 112], a claim in dependent form shall contain a reference to a claim previously set forth and then specify a further limitation of the subject matter claimed. A claim in dependent form shall be construed to incorporate by reference all the limitations of the claim to which it refers.
Claim 12 is rejected under 35 U.S.C. 112(d) or pre-AIA 35 U.S.C. 112, 4th paragraph, as being of improper dependent form for failing to further limit the subject matter of the claim upon which it depends, or for failing to include all the limitations of the claim upon which it depends. Claim 12 does not further limit the subject matter of claim 1 because the recited mobility system and its components are all identically recited in claim 1. Applicant may cancel the claim, amend the claim to place the claim in proper dependent form, rewrite the claim in independent form, or present a sufficient showing that the dependent claim complies with the statutory requirements.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-14 are rejected under 35 U.S.C. 103 as being unpatentable over US 20230333652 A1 (Nam et al.) in view of US 20200268296 A1 (Alcaide et al.).
Regarding claim 1, Nam teaches a method for controlling a mobility system (1), the mobility system comprising
an electric wheelchair (2) ([0022] “the mobility device 101 may be an electric powered wheelchair as shown in FIG. 1”);
an electroencephalogram, EEG, system (3) configured to detect steady state visual evoked potentials, SSVEPs, and to record EEG data ([0023] “The EEG device 110 may be configured to detect and/or measure SSVEP (Steady-State Visual Evoked Potential) of the user”);
a vision system (4) comprising a display (5), and the display (5) is configured to show visual stimuli and to enable a view of at least a part of the scene ([0052] “the projector 180 to output the visual stimulus on a surface at a position spaced from the mobility device 101 by a specified distance. The visual stimulus may be displayed at a position spaced by a certain distance away from the user's eyes”; [0065] “The modified visual stimulus 320 may be outputted from the projector 180 and may be displayed on the surface”);
a controller (7) configured to operatively communicate with and control the vision system and the electric wheelchair (2), and the controller (7) is also configured to operatively communicate with the EEG system (3) ([0022] “a controller 120”; [0023] “he EEG device 110 may be configured to transmit an EEG signal corresponding to a detected and/or measured SSVEP to the controller 120 (e.g., in real time as detected and/or measured).”; [0025] “Also, or alternatively, the controller 120 may be configured to control the projector 180 to output a visual stimulus (e.g., an image, such as a projected image).”);
wherein the method comprises the steps of:
a) the controller controlling the EEG system to record EEG data ([0023] “The EEG device 110 may be configured to transmit an EEG signal corresponding to a detected and/or measured SSVEP to the controller 120 (e.g., in real time as detected and/or measured).”);
b) the controller executing an algorithm to detect an object or path in the scene ([0032] “a surface (e.g., a floor, ground, wall, etc.) at a position spaced by a specified distance from a front of the mobility device 101 (e.g., in a direction that the wheelchair 101 is configured to have a user face when seated therein).”);
c) if in step (b) the object or path is detected, the controller controlling the display to show first visual stimuli within the view which is enabled by the display, the first visual stimuli overlapping or pointing towards the path or object ([0032]; [0052] “the controller 120 may control the projector 180 to output the visual stimulus on a surface at a position spaced from the mobility device 101 by a specified distance”; [0053] “The visual stimulus may comprise various objects (indications of direction, stop, speed change, etc.) for a variety of visual stimuli”);
d) the controller processing the EEG data to determine if first SSVEPs are generated in response to the first visual stimuli ([0032] “When a user gazes at one of the direction indication objects included in the visual stimulus for a duration greater than or equal to a specified time duration, a corresponding SSVEP may be generated by the user.”; [0042] “For example, when a user gazes at the turn-right indication 252 for 2 seconds or more, a corresponding first SSVEP may be measured via the EEG device 110. The EEG device 110 may generate a corresponding first EEG signal and transmit the first EEG signal to the controller 120. The controller 120 may receive the first EEG signal from the EEG device 110. The controller 120 may compare a preset reference value to one or more characteristics (e.g., an amplitude, a frequency, a length, etc.) of the first EEG signal. The controller 120 may generate a first control signal to control the drive 190 to perform a turn-right movement, based on the comparing result (e.g., if one or more characteristics of the first EEG signal characteristic are consistent with the preset reference value)”);
e) if in step (d) the controller determines that first SSVEPs are generated, the controller controlling the electric wheelchair to move towards the object or via said path ([0042] “The controller 120 may transmit the first control signal to the drive 190. The drive 190 may receive the first control signal and, based on the received first control signal, cause the mobility device 101 to turn to the right.”; [0056] “The controller 120 may transmit the generated control signal to the drive 190. The drive 190 may move, stop or turn the mobility device 101 in response to the received control signal.”).
Nam does not explicitly teach a camera (6) which is configured to capture a video or image of a scene, and the controller controlling the camera to capture the digital video or image of the scene, processing the digital video or image.
However,
Alcaide teaches a camera (6) which is configured to capture a video or image of a scene ([0033] “The video based eye-tracker can also include an additional scene camera that captures the user's field of view.”), and
the controller controlling the camera to capture the digital video or image of the scene, processing the digital video or image ([0086] “The UI/UX Engine 526 can use context based information provided by the user, the BCI Device 110 (510), or from the user's environment (relayed through a visual field camera) and generate/update the UI or UX presented with appropriate controls to mediate navigation.”; [0087]).
It would have been obvious for one of ordinary skill in the art before the effective filing date of the invention to have modified the method taught by Nam to include a camera. One would have been motivated to make this modification because a camera is able to provide information regarding the user’s environment that can allow for appropriate controls to mediate navigation of a wheelchair in combination with using EEG neural control signals, as suggested by Alcaide [0029, 0086].
Regarding claim 2, Nam teaches a method according to claim 1, wherein executing the algorithm in step (b) comprises using a neural network, preferably a convolutional neural network, for detecting the object or path in the scene ([0079] “the controller 120 may classify a direction-related intention of the user using a SSVEP recognition algorithm … The SSVEP recognition algorithm may employ an algorithm using deep learning techniques such as TRCA, CNN, and LSTM that requires training as needed, which may allow for increased accuracy and/or precision.”).
Regarding claim 3, Nam teaches a method according to claim 1, wherein the method further comprises the steps of:
f) the controller controlling the display to show second visual stimuli ([0025] “the controller 120 may be configured to control the projector 180 to output a visual stimulus (e.g., an image, such as a projected image).”; [0032] “The visual stimulus may comprise direction indications, such as go-forward, turn-left, and turn-right indication”; [0043]);
g) the controller processing the EEG data to determine if second SSVEPs are generated in response to the second visual stimuli ([0043] “when the user gazes at the turn-left indication object 253 for 2 seconds or more, a corresponding second SSVEP may be measured via the EEG device 110. The EEG device 110 may generate a corresponding second EEG signal and transmit the second EEG signal to the controller 120. The controller 120 may receive the second EEG signal from the EEG device 110. The controller 120 may compare a preset reference value and one or more characteristics of the second EEG signal with each other.”);
h) if in step (g) determining that the second SSVEPs are generated, the controller controlling the electric wheelchair to move forward, left or right ([0043] “The controller 120 may generate a second control signal to control the drive 190 to perform a turn-left movement, based on the comparing result. The controller 120 may transmit the second control signal to the drive 190. The drive 190 may receive the second control signal and, based on the received second control signal, cause the mobility device 101 to turn to the left.”).
Regarding claim 4, Nam teaches a method according to claim 3, wherein the first and second visual stimuli are graphics which flicker at the same or different frequencies with respect to each other ([0033] “the direction indications (e.g., go-forward, turn-left, turn-right indication objects) may be projected to flicker (e.g., to vary in brightness) at one or more frequencies. Also, or alternatively, the direction indications may be projected to vary temporarily in color, pattern, focus, and/or some other visual manner. Different direction indications may be projected to flicker at different frequencies.”).
Regarding claim 5, Nam teaches a method according to claim 3, wherein the second visual stimuli comprise three flickering checkerboards each of which capable of causing respective second SSVEPS for causing a respective one of the forward, left or right movement in step (h) ([0033] “the direction indications (e.g., go-forward, turn-left, turn-right indication objects) may be projected to flicker (e.g., to vary in brightness) at one or more frequencies. Also, or alternatively, the direction indications may be projected to vary temporarily in color, pattern, focus, and/or some other visual manner. Different direction indications may be projected to flicker at different frequencies.”; [0039] “each of the direction indication objects 251, 252, and 253 may have patterns of dark and light areas that may be varied over time, such as reversed over time. For example, the patterns may comprise a checker board pattern, and black (e.g., dark) and white (e.g., light) of the checker board pattern may be alternated with each other over time.”; [0053]; [0060], [0064], [0066]).
Regarding claim 6, Nam teaches a method according to claim 1, wherein the visual stimuli comprise flickering graphics, preferably flickering targets or checkerboards ([0033] “the direction indications (e.g., go-forward, turn-left, turn-right indication objects) may be projected to flicker (e.g., to vary in brightness) at one or more frequencies. Also, or alternatively, the direction indications may be projected to vary temporarily in color, pattern, focus, and/or some other visual manner. Different direction indications may be projected to flicker at different frequencies.”; [0039] “each of the direction indication objects 251, 252, and 253 may have patterns of dark and light areas that may be varied over time, such as reversed over time. For example, the patterns may comprise a checker board pattern, and black (e.g., dark) and white (e.g., light) of the checker board pattern may be alternated with each other over time.”; [0053]; [0060], [0064], [0066]).
Regarding claim 7, Nam teaches a method according to claim 1, wherein the display enables viewing a part of the scene, the object or path in step (a) is outside the part of the scene, and the first visual stimuli in step (b) comprise arrows pointing towards the object or path ([0032] “The visual stimulus may be projected on a surface (e.g., a floor, ground, wall, etc.) at a position spaced by a specified distance from a front of the mobility device 101 (e.g., in a direction that the wheelchair 101 is configured to have a user face when seated therein). The visual stimulus may comprise direction indications, such as go-forward, turn-left, and turn-right indication”; [0060] “each of the direction indications 311, 312, and 313 may be in a form of an undistorted arrow. Each of the direction indications 311, 312, and 313 may have a checker board pattern. In the basic visual stimulus 310, the direction indications 311, 312, and 313 may be centered vertically in an area of the visual stimulus.”).
Regarding claim 8, Nam teaches a method according to claim 1, wherein the display is augmented reality, AR, glasses, or is a monitor ([0006] “A BCI-based mobility device (or mobility device control system) using SSVEP may use an LED or LCD monitor for generating a visual stimulus to induce the SSVEP”)
Regarding claim 9, Nam teaches a method according to claim 1, wherein the method further comprises the following steps after step (b) and before step (c):
b1) if in step (b) the object or path is detected, the controller controlling the display or a speaker of the system to indicate the detection of the object or path ([0032] “The visual stimulus may be projected on a surface (e.g., a floor, ground, wall, etc.) at a position spaced by a specified distance from a front of the mobility device 101 (e.g., in a direction that the wheelchair 101 is configured to have a user face when seated therein). The visual stimulus may comprise direction indications, such as go-forward, turn-left, and turn-right indication. When a user gazes at one of the direction indication objects included in the visual stimulus for a duration greater than or equal to a specified time duration, a corresponding SSVEP may be generated by the user.”; [0062] “The controller 120 may change the basic visual stimulus 310 to a modified visual stimulus 320 and transmit information indicating the modified visual stimulus 320 to the projector 180. The modified visual stimulus 320 may be modified to reduce the distortion of a perceived visual stimulus 330 or a direction and/or motion indication in the perceived visual stimulus 330.”; [0064] “In the modified visual stimulus 320, the direction indications 321, 322, and 323 may be positioned in a lower area of the modified visual stimulus 320 in the vertical direction (relative to FIG. 3).”);
b2) the controller controlling the screen to display confirmation visual stimuli ([0062] “The controller 120 may change the basic visual stimulus 310 to a modified visual stimulus 320 and transmit information indicating the modified visual stimulus 320 to the projector 180. The modified visual stimulus 320 may be modified to reduce the distortion of a perceived visual stimulus 330 or a direction and/or motion indication in the perceived visual stimulus 330.”);
b3) the controller processing the EEG data to determine if confirmation SSVEPs are generated in response to the confirmation visual stimuli ([0055] “the controller 120 may generate a control signal corresponding to the brainwave detected via the EEG device 110. The controller 120 may receive a signal (e.g., corresponding to the measured and/or detected SSVEP) corresponding to the brainwave detected via the EEG device 110. The controller 120 may generate the control signal corresponding to the EEG signal received from the EEG device 110. For example, the control signal may comprise a go-straight signal, a turn-left signal, a turn-right signal, or a stop signal.”);
b4) if in step (b3) determining that the confirmation SSVEPs are generated in response to the confirmation visual stimuli, then the controller proceeds to execute step (c) ([0027] “The controller 120 and/or a component thereof (e.g., the stimulus sensor 121) may be configured to generate a control signal for controlling the drive 190 based on a received EEG signal (e.g., based on an analyzed result of a received EEG signal indicating a SSVEP).”).
Regarding claim 10, Nam teaches a method according to claim 9, wherein the controller determining if a condition is met for executing step (c), the condition being that the controller is not triggered to not execute step (c), and/or that the controller is triggered, preferably by means of executing steps (b1)-(b4), to execute step (c) ([0054] “the controller 120 may control the EEG device 110 to detect the user's brainwave induced by the visual stimulus. The EEG device 110 may be mountable on the user's head, and may be configured to measure the brainwaves (e.g., measure and/or detect SSVEP) generated when the user gazes at the visual stimulus (e.g., for a threshold amount of time).”; [0055]).
Regarding claim 11, Nam teaches a method according to claim 1, wherein in step (b) detecting an object, and the method further comprises:
the controller identifying the object ([0050]; [0051] “The controller 120 may detect a color and/or reflectivity of the surface around the mobility device 101. The controller may output the visual stimulus with a brightness, contrast, color hue and/or saturation to improve visibility on the surface.”);
the controller determining if according to the object's identity, a further condition is met ([0051] “For example, the visual stimulus may be projected in a color contrasting with the color of the bottom surface.”);
If the further condition is met, the controller controlling the display to show control visual stimuli, preferably a plethora of control visual stimuli related to respective commands for controlling the object ([0051] “The controller may output the visual stimulus with a brightness, contrast, color hue and/or saturation to improve visibility on the surface.”);
the controller processing the EEG data to determine if control SSVEPs are generated in response to the control visual stimuli ([0055] “the controller 120 may generate a control signal corresponding to the brainwave detected via the EEG device 110. The controller 120 may receive a signal (e.g., corresponding to the measured and/or detected SSVEP) corresponding to the brainwave detected via the EEG device 110.”);
if the controller determines that the control SSVEPs are generated, the controller generating a command signal for controlling the object ([0055] “The controller 120 may generate the control signal corresponding to the EEG signal received from the EEG device 110. For example, the control signal may comprise a go-straight signal, a turn-left signal, a turn-right signal, or a stop signal.”; [0050] “For example, the visual stimulus may be projected in a color contrasting with the color of the bottom surface.”; [0032] “The visual stimulus may be projected on a surface (e.g., a floor, ground, wall, etc.) at a position spaced by a specified distance from a front of the mobility device 10”).
Regarding claim 12, Nam teaches a mobility system comprising:
an electric wheelchair ([0022] “the mobility device 101 may be an electric powered wheelchair as shown in FIG. 1”);
an electroencephalogram, EEG, system configured to detect steady state visual evoked potentials, SSVEPs, and to record EEG data ([0023] “The EEG device 110 may be configured to detect and/or measure SSVEP (Steady-State Visual Evoked Potential) of the user”);
a vision system comprising a display and a camera which is configured to capture a video or image of a scene, and the display is configured to show visual stimuli and to enable a view of at least a part of the scene ([0052] “the projector 180 to output the visual stimulus on a surface at a position spaced from the mobility device 101 by a specified distance. The visual stimulus may be displayed at a position spaced by a certain distance away from the user's eyes”; [0065] “The modified visual stimulus 320 may be outputted from the projector 180 and may be displayed on the surface”);
a controller configured to operatively communicate with and control the vision system and the electric wheelchair, and the controller is also configured to operatively communicate with the EEG system ([0022] “a controller 120”; [0023] “he EEG device 110 may be configured to transmit an EEG signal corresponding to a detected and/or measured SSVEP to the controller 120 (e.g., in real time as detected and/or measured).”; [0025] “Also, or alternatively, the controller 120 may be configured to control the projector 180 to output a visual stimulus (e.g., an image, such as a projected image).”);
wherein the controller is adapted to execute the steps of the method of claim 1 ([0055] “the controller 120 may generate a control signal corresponding to the brainwave detected via the EEG device 110. The controller 120 may receive a signal (e.g., corresponding to the measured and/or detected SSVEP) corresponding to the brainwave detected via the EEG device 11”).
Regarding claim 13, Nam teaches a controller for a mobility system, the controller comprising:
a processor (701) ([0025] “controller 120 (e.g., a processor)”; [0083] “at least one processor 1100”);
a memory (702) ([0083] “a memory 1300,”);
one or more interfaces (703) for operatively communicating and controlling an electric wheelchair, an electroencephalogram, EEG, system, and a vision system, wherein the vision comprises a display, the display is configured to show visual stimuli, the EEG system is configured to detect steady state visual evoked potentials, SSVEPs, and to record EEG data ([0022] “a mobility device 101 may include an EEG device 110, a controller 120, a projector 180, and a drive 190 (e.g., including motor(s), and/or any other motion controller(s) to provide motion to, or change motion of, the mobility device, such as the mobility device 101, or a part thereof). While the mobility device 101 may be an electric powered wheelchair as shown in FIG. 1 or another device (e.g., a scooter, a car, a bike, the mobility device with treads, legs, etc.) configured to be controlled by a drive, such as the drive 190, could be used.”);
and wherein the controller further comprises
an EEG control module (704) for controlling the EEG system to record EEG data ([0023] “The EEG device 110 may be configured to transmit an EEG signal corresponding to a detected and/or measured SSVEP to the controller 120 (e.g., in real time as detected and/or measured).”);
an image processing module (706) for processing and executing an algorithm to detect an object or path in the scene ([0032] “a surface (e.g., a floor, ground, wall, etc.) at a position spaced by a specified distance from a front of the mobility device 101 (e.g., in a direction that the wheelchair 101 is configured to have a user face when seated therein).”);
a display control module for controlling the display to show first visual stimuli within the view which is enabled by the display, the first visual stimuli overlapping or pointing towards the path or object ([0032]; [0052] “the controller 120 may control the projector 180 to output the visual stimulus on a surface at a position spaced from the mobility device 101 by a specified distance”; [0053] “The visual stimulus may comprise various objects (indications of direction, stop, speed change, etc.) for a variety of visual stimuli”);
an EEG processing module (707) for processing the EEG data to determine if SSVEPs are generated in response to the corresponding visual stimuli ([0032] “When a user gazes at one of the direction indication objects included in the visual stimulus for a duration greater than or equal to a specified time duration, a corresponding SSVEP may be generated by the user.”; [0042] “For example, when a user gazes at the turn-right indication 252 for 2 seconds or more, a corresponding first SSVEP may be measured via the EEG device 110. The EEG device 110 may generate a corresponding first EEG signal and transmit the first EEG signal to the controller 120. The controller 120 may receive the first EEG signal from the EEG device 110. The controller 120 may compare a preset reference value to one or more characteristics (e.g., an amplitude, a frequency, a length, etc.) of the first EEG signal. The controller 120 may generate a first control signal to control the drive 190 to perform a turn-right movement, based on the comparing result (e.g., if one or more characteristics of the first EEG signal characteristic are consistent with the preset reference value)”);
an electric wheelchair control module (708) for controlling the electric wheelchair to move towards the object, or via the path, which is detected with the image processing module ([0042] “The controller 120 may transmit the first control signal to the drive 190. The drive 190 may receive the first control signal and, based on the received first control signal, cause the mobility device 101 to turn to the right.”; [0056] “The controller 120 may transmit the generated control signal to the drive 190. The drive 190 may move, stop or turn the mobility device 101 in response to the received control signal.”).
Nam does not explicitly teach and a camera which is configured to capture a video or image of a scene, a camera control module (705) for controlling the camera to capture the digital video or image of the scene, and an image processing module (706) for processing the digital video or image,
However,
Alcaide teaches and a camera which is configured to capture a video or image of a scene ([0033] “The video based eye-tracker can also include an additional scene camera that captures the user's field of view.”),
a camera control module (705) for controlling the camera to capture the digital video or image of the scene [0086] “The UI/UX Engine 526 can use context based information provided by the user, the BCI Device 110 (510), or from the user's environment (relayed through a visual field camera)”), and
an image processing module (706) for processing the digital video or image ([0086] “The UI/UX Engine 526 can use context based information provided by the user, the BCI Device 110 (510), or from the user's environment (relayed through a visual field camera) and generate/update the UI or UX presented with appropriate controls to mediate navigation.”; [0087]).
It would have been obvious for one of ordinary skill in the art before the effective filing date of the invention to have modified the controller taught by Nam to include a camera. One would have been motivated to make this modification because a camera is able to provide information regarding the user’s environment that can allow for appropriate controls to mediate navigation of a wheelchair in combination with using EEG neural control signals, as suggested by Alcaide [0029, 0086].
Regarding claim 14, Nam teaches a controller according to claim 13, wherein the image processing module comprises a neural network, preferably a convolutional neural network ([0079] “the controller 120 may classify a direction-related intention of the user using a SSVEP recognition algorithm … The SSVEP recognition algorithm may employ an algorithm using deep learning techniques such as TRCA, CNN, and LSTM that requires training as needed, which may allow for increased accuracy and/or precision.”).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to EVELYN GRACE PARK whose telephone number is (571)272-0651. The examiner can normally be reached Monday - Friday, 9AM - 5:00PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Robert (Tse) Chen can be reached at (571)272-3672. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/EVELYN GRACE PARK/Examiner, Art Unit 3791 /TSE CHEN/Supervisory Patent Examiner, Art Unit 3791
Prosecution Insights