DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Response to Amendment
The following is an office action in response to the communication filed on 09/25/2025.
Claim(s) 1-10, 12, 17 is/are amended.
Claim(s) 11, 13-16 is/are original.
Claim(s) 18-20 is/are new.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 10/07/2025 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 3, 11, and 19 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 3 line 3 refers to "a video feed from a camera recording device". It is unclear if this video feed is the same video data as presented in claim 1 line 10 or a different video feed. Additionally, it is unclear if this camera recording device is the same camera directed towards the operator as presented in claim 1 line 10 or a different camera.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-3, 5, 7-11, 13, 15-17, and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 9925921 B1 Deyaf; Hamza et al. (hereinafter Deyaf), in view of US 20150309569 A1 Kohlhoff; Kai Jochen et al. (hereinafter Kohlhoff), further in view of US 20210229673 A1 Singh; Hanumant Prasad R et al. (hereinafter Singh), and further in view of CN 108089695 A WANG, HAN et al. (hereinafter Wang).
Regarding claim 1, Deyaf teaches: A computer-implemented method (see Deyaf at least [col. 1, lines 8-9] method for controlling a direction and orientation of a light source) comprising:
generating, (see Deyaf at least [col. 6, lines 9-15] user controller 106 includes a wireless transmitting device that may wirelessly communicate a position of a user's head 122, including the horizontal and/or vertical position of a user's head 122 to auxiliary vehicle light 108 so that auxiliary vehicle light 108 is directed at the same angle and position as the user's head 122); and
actuating at least one motor of the auxiliary device to move based on the instructions (see Deyaf at least [col. 17, lines 37-42] Upon receiving the coordinates of user's head 424 (e.g. indicating a displacement or change in position), microcontroller 408 in one or more embodiments may send instructions to one or more motor drivers 410 to actuate one or more motors 412 coupled to pivoting device 414).
Deyaf does not teach: receiving, from a camera directed towards an operator, one or more images associated with the operator, wherein the one or more images include one or more features of the operator, and generating, using video data from the camera directed towards the operator, a representation of a movement associated with the operator by tracking one or more facial landmarks associated with the operator included in the video data, wherein the representation of the movement includes a first set of one or more two-dimensional vectors.
However, Kohlhoff teaches: receiving, from a camera directed towards an operator, one or more images associated with the operator, wherein the one or more images include one or more features of the operator (see Kohlhoff at least [0029] The camera 116 can capture still images and/or video of a user 105 and [0061] the image may be extracted from a video feed received from a camera, such as the camera 116 of FIG. 1. The image may include the face of a user), and
generating, using video data from the camera directed towards the operator, a representation of a movement associated with the operator by tracking one or more facial landmarks associated with the operator included in the video data, wherein the representation of the movement includes a first set of one or more two-dimensional vectors (see Kohlhoff at least [0048] the gaze tracker 134 may generate a vector of the user's viewing direction based on the orientation of the user's face and the location of the user's face with respect to the displays 214 and 224. The gaze tracker 134 may then project the vector forward from the user's face to determine whether the user 205 is viewing one of the displays 214 and 224. The gaze tracker 134 can also use the vector to determine what portion or point on the display 214 or 224 the user 205 is viewing).
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the movable device controlling method disclosed by Deyaf to include the user facial direction tracking of Kohlhoff. One of ordinary skill in the art would have been motivated to make this modification because identifying the direction of a user’s gaze allows for control based on the gaze, limiting distractions in the vehicle, as suggested by Kohlhoff (see Kohlhoff at least [0012] User interface elements can be controlled based on a user's viewing direction and/or gestures, reducing the need for touch-based control without distracting others).
Deyaf and Kohlhoff do not teach: comparing the one or more features of the operator to one or more stored identities associated with an auxiliary device, wherein the one or more stored identities are associated with respective features: and identifying, based on the comparison of the one or more features to the one or more stored identities, a matching identity associated with the operator.
However, Singh teaches: comparing the one or more features of the operator to one or more stored identities associated with an auxiliary device, wherein the one or more stored identities are associated with respective features (see Singh at least [0007] after determining the presence of the user inside the vehicle, capture, using an image capture device that is connected to at least a portion of the vehicle, at least one image of a face of the user; determine first feature data associated with the at least one image of the face of the user; receive authentication data for the user, wherein the authentication data indicates a match between the user and a previously enrolled user based on a comparison between the first feature data and second feature data associated with at least one image of a face of the previously enrolled user): and
identifying, based on the comparison of the one or more features to the one or more stored identities, a matching identity associated with the operator (see Singh at least [0053] Mobile computing device 170 has authenticated user 150 based on a comparison of received facial feature data with stored facial feature data of authorized, known users stored by mobile computing device 170, to identify a match with the stored facial feature data of user 150).
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the movable device controlling method disclosed by Deyaf and Kohlhoff to include the image-based vehicle occupant authentication of Singh. One of ordinary skill in the art would have been motivated to make this modification because the hands-free authentication technique provides an easy path for users to access their user-specific vehicle preferences, as suggested by Singh (see Singh at least [0002] various of the described techniques enable a seamless and reliable authentication of a driver of a vehicle).
Deyaf, Kohlhoff, and Singh do not teach: receiving an identification of a location from the auxiliary device that indicates a current location of the auxiliary device, wherein the identification includes a second set of one or more two-dimensional vectors; comparing the first set of one or more two-dimensional vectors to the second set of one or more two-dimensional vectors: and generating, based on the comparison of the first set of one or more two-dimensional vectors to the second set of one or more two-dimensional vectors, instructions for the auxiliary device to perform an auxiliary movement.
However, Wang teaches: receiving an identification of a location from the auxiliary device that indicates a current location of the auxiliary device, wherein the identification includes a second set of one or more two-dimensional vectors (see Wang at least [0083] For the second motion vector, the current motion direction and speed of the movable device, as well as the distance and direction between the movable device and the target are collected, and then the second motion vector is synthesized);
comparing the first set of one or more two-dimensional vectors to the second set of one or more two-dimensional vectors (see Wang at least [0122] A superposition module 102, configured to superimpose the first motion description information corresponding to the first human posture and the second motion description information currently executed by the movable device to obtain third motion description information and [0061] the motion description information in the embodiment of the present invention is information used to represent motion, such as motion vector): and
generating, based on the comparison of the first set of one or more two-dimensional vectors to the second set of one or more two-dimensional vectors, instructions for the auxiliary device to perform an auxiliary movement (see Wang at least [0058] Calculate the target movement parameter corresponding to the third motion description information, and control the movable device to move with the target movement parameter).
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the movable device controlling method disclosed by Deyaf, Kohlhoff, and Singh to include the vector superimposition to generate auxiliary device motion control of Wang. One of ordinary skill in the art would have been motivated to make this modification because controlling a device based on the orientation of a user allows for simple hands-free control, as suggested by Wang (see Wang at least [0135] the user can control the movement of the movable device by making different postures, without adding an additional controller).
Regarding claim 2, Deyaf, Kohlhoff, Singh, and Wang disclose: The computer-implemented method of claim 1, wherein the representation of the movement associated with the operator is based on the one or more facial landmarks of the operator (see Kohlhoff at least [0038] The gaze tracker 134 may generate a template image for each facial feature point of the user's face and [0039] The gaze tracker 134 can use the template images to determine the orientation of the user's face in other images and [0042] the gaze tracker 134 may generate a vector of the user's viewing direction based on the orientation and the location of the user's face).
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the movable device controlling method disclosed by Deyaf, Kohlhoff, Singh, and Wang to include the facial landmark consideration of Kohlhoff. One of ordinary skill in the art would have been motivated to make this modification because identifying feature points of a user’s face allows the direction of the user’s gaze to be used as an input for an interface system, as suggested by Kohlhoff (see Kohlhoff at least [0012] User interface elements can be controlled based on a user's viewing direction and/or gestures, reducing the need for touch-based control without distracting others).
Regarding claim 3, Deyaf, Kohlhoff, Singh, and Wang disclose: The computer-implemented method of claim 2, further comprising: receiving a video feed from a camera recording device, wherein the video feed contains a human figure (see Kohlhoff at least [0033] the gaze-based UI control module 130 may receive a video feed from the camera 116 and [0029] The camera 116 can capture still images and/or video of a user 105);
identifying the human figure in the video feed, wherein the human figure is the operator (see Kohlhoff at least [0041] If the gaze tracker 134 finds a matching image portion for each template image, or at least a threshold number or percentage of template images, the gaze tracker 134 may determine that the user's face is depicted in the image);
identifying, on the human figure, the one or more facial landmarks (see Kohlhoff at least [0041] if the relative locations of the matching image portions indicate that the distance between the left side of the user's face and the tip of the user's nose is less than the distance between the right side of the user's face and the tip of the user's nose, then the gaze tracker 134 may determine that the user 105 is looking to the left.); and
tracking, through the video feed, the one or more facial landmarks (see Kohlhoff at least [0066] The fitted face model can be used to quickly redetect the face and track the face in subsequent images).
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the movable device controlling method disclosed by Deyaf, Kohlhoff, Singh, and Wang to include the facial landmark consideration of Kohlhoff. One of ordinary skill in the art would have been motivated to make this modification because tracking feature points of a user’s face allows the direction of the user’s gaze to be used as an input for an interface system, as suggested by Kohlhoff (see Kohlhoff at least [0012] User interface elements can be controlled based on a user's viewing direction and/or gestures, reducing the need for touch-based control without distracting others).
Regarding claim 5, Deyaf, Kohlhoff, Singh, and Wang disclose: The computer-implemented method of claim 1, further comprising: setting an operator reference vector for the operator (see Deyaf at least [col. 11, lines 23-27] Set of sensors 446 may be configured to sense displacement and position of user's head 424 along one, two, or three axes, including a horizontal and vertical axis that corresponds to a horizontal and vertical axis aligned with a user's head 422); and
calibrating the representation of the movement associated with the operator, wherein calibrating the representation of the movement comprises of comparing the operator reference vector to the first set of one or more two-dimensional vectors (see Deyaf at least [col. 11, lines 29-37] One or more sensors of set of sensors 446 may be described as an angular position sensor… An angular position sensor calculates the orientation of an object with respect to a specified reference position s expressed by the amount of rotation necessary to change from one orientation to the other about a specified axis).
Regarding claim 7, Deyaf, Kohlhoff, Singh, and Wang disclose: The computer-implemented method of claim 1, wherein the instructions for the auxiliary device to perform the auxiliary movement are inputted manually (see Deyaf at least [col. 16, lines 56-58] there may be a selector such as input selector 470 located on user controller 432 that enables powering on and off auxiliary vehicle light 460. Upon powering on auxiliary vehicle light 460 (e.g. using any of the above), light output 420 may be emitted from auxiliary vehicle light 460, whereby a set of light beams are automatically projected from auxiliary vehicle light 460).
Regarding claim 8, Deyaf, Kohlhoff, Singh, and Wang disclose: The computer-implemented method of claim 1, wherein the auxiliary device is comprised of one or more electric motors (see Deyaf at least [col. 19, line 67 – col. 20, line 2] auxiliary vehicle light 902 may include one or more motors configured to pan or tilt (or roll) auxiliary vehicle light 902 about one or more axes).
Regarding claim 9, Deyaf discloses: A system (see Deyaf at least [col. 1, lines 7-9) comprising:
one or more processors (see Deyaf at least [col. 11, lines 64-65] a processor); and
a non-transitory computer-readable medium (see Deyaf at least [col. 12, line 9] flash size, RAM size) storing instructions that, when executed by the one or more processors, cause the one or more processors to:
generate(see Deyaf at least [col. 6, lines 9-15] user controller 106 includes a wireless transmitting device that may wirelessly communicate a position of a user's head 122, including the horizontal and/or vertical position of a user's head 122 to auxiliary vehicle light 108 so that auxiliary vehicle light 108 is directed at the same angle and position as the user's head 122); and
actuate at least one motor of the auxiliary device to move based on the instructions (see Deyaf at least [col. 17, lines 37-42] Upon receiving the coordinates of user's head 424 (e.g. indicating a displacement or change in position), microcontroller 408 in one or more embodiments may send instructions to one or more motor drivers 410 to actuate one or more motors 412 coupled to pivoting device 414).
Deyaf does not teach: receive, from a camera directed towards an operator, one or more images associated with the operator, wherein the one or more images include one or more features of the operator: and generate, using video data from the camera directed towards the operator, a representation of a movement associated with the operator by tracking one or more facial landmarks associated with the operator included in the video data, wherein the representation of the movement includes a first set of one or more two- dimensional vectors.
However, Kohlhoff teaches: receive, from a camera directed towards an operator, one or more images associated with the operator, wherein the one or more images include one or more features of the operator (see Kohlhoff at least [0029] The camera 116 can capture still images and/or video of a user 105 and [0061] the image may be extracted from a video feed received from a camera, such as the camera 116 of FIG. 1. The image may include the face of a user): and
generate, using video data from the camera directed towards the operator, a representation of a movement associated with the operator by tracking one or more facial landmarks associated with the operator included in the video data, wherein the representation of the movement includes a first set of one or more two- dimensional vectors (see Kohlhoff at least [0048] the gaze tracker 134 may generate a vector of the user's viewing direction based on the orientation of the user's face and the location of the user's face with respect to the displays 214 and 224. The gaze tracker 134 may then project the vector forward from the user's face to determine whether the user 205 is viewing one of the displays 214 and 224. The gaze tracker 134 can also use the vector to determine what portion or point on the display 214 or 224 the user 205 is viewing).
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the movable device controlling system disclosed by Deyaf to include the user facial direction tracking of Kohlhoff. One of ordinary skill in the art would have been motivated to make this modification because identifying the direction of a user’s gaze allows for control based on the gaze, limiting distractions in the vehicle, as suggested by Kohlhoff (see Kohlhoff at least [0012] User interface elements can be controlled based on a user's viewing direction and/or gestures, reducing the need for touch-based control without distracting others).
Deyaf and Kohlhoff do not teach: compare the one or more features of the operator to one or more stored identities associated with an auxiliary device, wherein the one or more stored identities are associated with respective features: and identify, based on the comparison of the one or more features to the one or more stored identities, a matching identity associated with the operator.
However, Singh teaches: compare the one or more features of the operator to one or more stored identities associated with an auxiliary device, wherein the one or more stored identities are associated with respective features (see Singh at least [0007] after determining the presence of the user inside the vehicle, capture, using an image capture device that is connected to at least a portion of the vehicle, at least one image of a face of the user; determine first feature data associated with the at least one image of the face of the user; receive authentication data for the user, wherein the authentication data indicates a match between the user and a previously enrolled user based on a comparison between the first feature data and second feature data associated with at least one image of a face of the previously enrolled user): and
identify, based on the comparison of the one or more features to the one or more stored identities, a matching identity associated with the operator (see Singh at least [0053] Mobile computing device 170 has authenticated user 150 based on a comparison of received facial feature data with stored facial feature data of authorized, known users stored by mobile computing device 170, to identify a match with the stored facial feature data of user 150).
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the movable device controlling system disclosed by Deyaf and Kohlhoff to include the image-based vehicle occupant authentication of Singh. One of ordinary skill in the art would have been motivated to make this modification because the hands-free authentication technique provides an easy path for users to access their user-specific vehicle preferences, as suggested by Singh (see Singh at least [0002] various of the described techniques enable a seamless and reliable authentication of a driver of a vehicle).
Deyaf, Kohlhoff, and Singh do not teach: receive an identification of a location from the auxiliary device that indicates a current location of the auxiliary device, wherein the identification includes a second set of one or more two-dimensional vectors; and compare the first set of one or more two-dimensional vectors to the second set of one or more two-dimensional vectors; and generating, based on the comparison of the first set of one or more two-dimensional vectors to the second set of one or more two-dimensional vectors, instructions for the auxiliary device to perform an auxiliary movement.
However, Wang teaches: receive an identification of a location from the auxiliary device that indicates a current location of the auxiliary device, wherein the identification includes a second set of one or more two-dimensional vectors (see Wang at least [0083] For the second motion vector, the current motion direction and speed of the movable device, as well as the distance and direction between the movable device and the target are collected, and then the second motion vector is synthesized);
compare the first set of one or more two-dimensional vectors to the second set of one or more two-dimensional vectors (see Wang at least [0122] A superposition module 102, configured to superimpose the first motion description information corresponding to the first human posture and the second motion description information currently executed by the movable device to obtain third motion description information and [0061] the motion description information in the embodiment of the present invention is information used to represent motion, such as motion vector); and
generating, based on the comparison of the first set of one or more two-dimensional vectors to the second set of one or more two-dimensional vectors, instructions for the auxiliary device to perform an auxiliary movement (see Wang at least [0058] Calculate the target movement parameter corresponding to the third motion description information, and control the movable device to move with the target movement parameter).
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the movable device controlling system disclosed by Deyaf, Kohlhoff, and Singh to include the vector superimposition to generate auxiliary device motion control of Wang. One of ordinary skill in the art would have been motivated to make this modification because controlling a device based on the orientation of a user allows for simple hands-free control, as suggested by Wang (see Wang at least [0135] the user can control the movement of the movable device by making different postures, without adding an additional controller).
Regarding claim 10, Deyaf, Kohlhoff, Singh, and Wang disclose: The system of claim 9, wherein the representation of the movement associated with the operator is based on the one or more facial landmarks of the operator (see Kohlhoff at least [0038] The gaze tracker 134 may generate a template image for each facial feature point of the user's face and [0039] The gaze tracker 134 can use the template images to determine the orientation of the user's face in other images and [0042] the gaze tracker 134 may generate a vector of the user's viewing direction based on the orientation and the location of the user's face).
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the movable device controlling method disclosed by Deyaf, Kohlhoff, Singh, and Wang to include the facial landmark consideration of Kohlhoff. One of ordinary skill in the art would have been motivated to make this modification because identifying feature points of a user’s face allows the direction of the user’s gaze to be used as an input for an interface system, as suggested by Kohlhoff (see Kohlhoff at least [0012] User interface elements can be controlled based on a user's viewing direction and/or gestures, reducing the need for touch-based control without distracting others).
Regarding claim 11, Deyaf, Kohlhoff, Singh, and Wang disclose: The system of claim 10, wherein the instructions further cause the one or more processors to:
receive a video feed from a camera recording device, wherein the video feed contains a human figure (see Kohlhoff at least [0033] the gaze-based UI control module 130 may receive a video feed from the camera 116 and [0029] The camera 116 can capture still images and/or video of a user 105);
identify the human figure in the video feed, wherein the human figure is the operator (see Kohlhoff at least [0041] If the gaze tracker 134 finds a matching image portion for each template image, or at least a threshold number or percentage of template images, the gaze tracker 134 may determine that the user's face is depicted in the image);
identify, on the human figure, the one or more facial landmarks (see Kohlhoff at least [0041] if the relative locations of the matching image portions indicate that the distance between the left side of the user's face and the tip of the user's nose is less than the distance between the right side of the user's face and the tip of the user's nose, then the gaze tracker 134 may determine that the user 105 is looking to the left.); and
track, through the video feed, the one or more facial landmarks (see Kohlhoff at least [0066] The fitted face model can be used to quickly redetect the face and track the face in subsequent images).
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the movable device controlling system disclosed by Deyaf, Kohlhoff, Singh, and Wang to include the facial landmark consideration of Kohlhoff. One of ordinary skill in the art would have been motivated to make this modification because tracking feature points of a user’s face allows the direction of the user’s gaze to be used as an input for an interface system, as suggested by Kohlhoff (see Kohlhoff at least [0012] User interface elements can be controlled based on a user's viewing direction and/or gestures, reducing the need for touch-based control without distracting others).
Regarding claim 13, Deyaf, Kohlhoff, Singh, and Wang disclose: The system of claim 9, wherein the instructions further cause the one or more processors to: set an operator reference vector for the operator (see Deyaf at least [col. 11, lines 23-27] Set of sensors 446 may be configured to sense displacement and position of user's head 424 along one, two, or three axes, including a horizontal and vertical axis that corresponds to a horizontal and vertical axis aligned with a user's head 422); and
calibrate the representation of the movement associated with the operator, wherein calibrating the representation of the movement comprises of comparing the operator reference vector to the first set of one or more two-dimensional vectors (see Deyaf at least [col. 11, lines 29-37] One or more sensors of set of sensors 446 may be described as an angular position sensor… An angular position sensor calculates the orientation of an object with respect to a specified reference position s expressed by the amount of rotation necessary to change from one orientation to the other about a specified axis).
Regarding claim 15, Deyaf, Kohlhoff, Singh, and Wang disclose: The system of claim 9, wherein the instructions for the auxiliary device to perform the auxiliary movement are inputted manually (see Deyaf at least [col. 16, lines 56-58] there may be a selector such as input selector 470 located on user controller 432 that enables powering on and off auxiliary vehicle light 460. Upon powering on auxiliary vehicle light 460 (e.g. using any of the above), light output 420 may be emitted from auxiliary vehicle light 460, whereby a set of light beams are automatically projected from auxiliary vehicle light 460).
Regarding claim 16, Deyaf, Kohlhoff, Singh, and Wang disclose: The system of claim 9, wherein the auxiliary device is comprised of one or more electric motors (see Deyaf at least [col. 19, line 67 – col. 20, line 2] auxiliary vehicle light 902 may include one or more motors configured to pan or tilt (or roll) auxiliary vehicle light 902 about one or more axes).
Regarding claim 17, Deyaf discloses: A non-transitory computer-readable storage medium (see Deyaf at least [col. 12, line 9] flash… RAM) storing instructions that, when executed by one or more processors (see Deyaf at least [col. 11, lines 64-65] a processor), cause the one or more processors to:
generate, (see Deyaf at least [col. 6, lines 9-15] user controller 106 includes a wireless transmitting device that may wirelessly communicate a position of a user's head 122, including the horizontal and/or vertical position of a user's head 122 to auxiliary vehicle light 108 so that auxiliary vehicle light 108 is directed at the same angle and position as the user's head 122); and
actuate at least one motor of the auxiliary device to move based on the instructions (see Deyaf at least [col. 17, lines 37-42] Upon receiving the coordinates of user's head 424 (e.g. indicating a displacement or change in position), microcontroller 408 in one or more embodiments may send instructions to one or more motor drivers 410 to actuate one or more motors 412 coupled to pivoting device 414).
Deyaf does not teach: receive, from a camera directed towards an operator, one or more images associated with the operator, wherein the one or more images include one or more features of the operator; and generate, using video data from the camera directed towards the operator, a representation of a movement associated with the operator by tracking one or more facial landmarks associated with the operator included in the video data, wherein the representation of the movement includes a first set of one or more two-dimensional vectors.
However, Kohlhoff teaches: receive, from a camera directed towards an operator, one or more images associated with the operator, wherein the one or more images include one or more features of the operator (see Kohlhoff at least [0029] The camera 116 can capture still images and/or video of a user 105 and [0061] the image may be extracted from a video feed received from a camera, such as the camera 116 of FIG. 1. The image may include the face of a user); and
generate, using video data from the camera directed towards the operator, a representation of a movement associated with the operator by tracking one or more facial landmarks associated with the operator included in the video data, wherein the representation of the movement includes a first set of one or more two-dimensional vectors (see Kohlhoff at least [0048] the gaze tracker 134 may generate a vector of the user's viewing direction based on the orientation of the user's face and the location of the user's face with respect to the displays 214 and 224. The gaze tracker 134 may then project the vector forward from the user's face to determine whether the user 205 is viewing one of the displays 214 and 224. The gaze tracker 134 can also use the vector to determine what portion or point on the display 214 or 224 the user 205 is viewing).
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the movable device controlling computer-readable medium disclosed by Deyaf to include the user facial direction tracking of Kohlhoff. One of ordinary skill in the art would have been motivated to make this modification because identifying the direction of a user’s gaze allows for control based on the gaze, limiting distractions in the vehicle, as suggested by Kohlhoff (see Kohlhoff at least [0012] User interface elements can be controlled based on a user's viewing direction and/or gestures, reducing the need for touch-based control without distracting others).
Deyaf and Kohlhoff do not teach: compare the one or more features of the operator to one or more stored identities associated with an auxiliary device, wherein the one or more stored identities are associated with respective features: and identify, based on the comparison of the one or more features to the one or more stored identities, a matching identity associated with the operator:
However, Singh teaches: compare the one or more features of the operator to one or more stored identities associated with an auxiliary device, wherein the one or more stored identities are associated with respective features (see Singh at least [0007] after determining the presence of the user inside the vehicle, capture, using an image capture device that is connected to at least a portion of the vehicle, at least one image of a face of the user; determine first feature data associated with the at least one image of the face of the user; receive authentication data for the user, wherein the authentication data indicates a match between the user and a previously enrolled user based on a comparison between the first feature data and second feature data associated with at least one image of a face of the previously enrolled user): and
identify, based on the comparison of the one or more features to the one or more stored identities, a matching identity associated with the operator (see Singh at least [0053] Mobile computing device 170 has authenticated user 150 based on a comparison of received facial feature data with stored facial feature data of authorized, known users stored by mobile computing device 170, to identify a match with the stored facial feature data of user 150).
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the movable device controlling method disclosed by Deyaf and Kohlhoff to include the image-based vehicle occupant authentication of Singh. One of ordinary skill in the art would have been motivated to make this modification because the hands-free authentication technique provides an easy path for users to access their user-specific vehicle preferences, as suggested by Singh (see Singh at least [0002] various of the described techniques enable a seamless and reliable authentication of a driver of a vehicle).
Deyaf, Kohlhoff, and Singh do not teach: receive an identification of a location from the auxiliary device that indicates a current location of the auxiliary device, wherein the identification includes a second set of one or more two-dimensional vectors: compare the first set of one or more two-dimensional vectors to the second set of one or more two-dimensional vectors: and generate, based on the comparison of the first set of one or more two-dimensional vectors to the second set of one or more two-dimensional vectors, instructions for the auxiliary device to perform an auxiliary movement.
However, Wang teaches: receive an identification of a location from the auxiliary device that indicates a current location of the auxiliary device, wherein the identification includes a second set of one or more two-dimensional vectors (see Wang at least [0083] For the second motion vector, the current motion direction and speed of the movable device, as well as the distance and direction between the movable device and the target are collected, and then the second motion vector is synthesized):
compare the first set of one or more two-dimensional vectors to the second set of one or more two-dimensional vectors (see Wang at least [0122] A superposition module 102, configured to superimpose the first motion description information corresponding to the first human posture and the second motion description information currently executed by the movable device to obtain third motion description information and [0061] the motion description information in the embodiment of the present invention is information used to represent motion, such as motion vector): and
generate, based on the comparison of the first set of one or more two-dimensional vectors to the second set of one or more two-dimensional vectors, instructions for the auxiliary device to perform an auxiliary movement (see Wang at least [0058] Calculate the target movement parameter corresponding to the third motion description information, and control the movable device to move with the target movement parameter).
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the movable device controlling method disclosed by Deyaf, Kohlhoff, and Singh to include the vector superimposition to generate auxiliary device motion control of Wang. One of ordinary skill in the art would have been motivated to make this modification because controlling a device based on the orientation of a user allows for simple hands-free control, as suggested by Wang (see Wang at least [0135] the user can control the movement of the movable device by making different postures, without adding an additional controller).
Regarding claim 19, Deyaf, Kohlhoff, Singh, and Wang disclose: The non-transitory computer-readable storage medium of claim 17, wherein the instructions further cause the one or more processors to: receive a video feed from a camera recording device, wherein the video feed contains a human figure (see Kohlhoff at least [0033] the gaze-based UI control module 130 may receive a video feed from the camera 116 and [0029] The camera 116 can capture still images and/or video of a user 105);
identify the human figure in the video feed, wherein the human figure is the operator (see Kohlhoff at least [0041] If the gaze tracker 134 finds a matching image portion for each template image, or at least a threshold number or percentage of template images, the gaze tracker 134 may determine that the user's face is depicted in the image);
identify, on the human figure, the one or more facial landmarks (see Kohlhoff at least [0041] if the relative locations of the matching image portions indicate that the distance between the left side of the user's face and the tip of the user's nose is less than the distance between the right side of the user's face and the tip of the user's nose, then the gaze tracker 134 may determine that the user 105 is looking to the left); and
track, through the video feed, the one or more facial landmarks (see Kohlhoff at least [0066] The fitted face model can be used to quickly redetect the face and track the face in subsequent images).
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the movable device controlling computer-readable medium disclosed by Deyaf, Kohlhoff, Singh, and Wang to include the facial landmark consideration of Kohlhoff. One of ordinary skill in the art would have been motivated to make this modification because tracking feature points of a user’s face allows the direction of the user’s gaze to be used as an input for an interface system, as suggested by Kohlhoff (see Kohlhoff at least [0012] User interface elements can be controlled based on a user's viewing direction and/or gestures, reducing the need for touch-based control without distracting others).
Claim(s) 4, 12, and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Deyaf, in view of Kohlhoff, further in view of Singh, further in view of Wang, and further in view of US 20160224852 A1 Vicente; Francisco et al. (hereinafter Vicente).
Regarding claim 4, Deyaf, Kohlhoff, Singh, and Wang disclose: The computer-implemented method of claim 2.
Deyaf, Kohlhoff, Singh, and Wang do not disclose: Further comprising: converting the one or more facial landmarks to one or more two-dimensional and three-dimensional points; translating the one or more two-dimensional and three-dimensional points to one or more three-dimensional vectors; and converting the one or more three-dimensional vectors to a rotational matrix.
However, Vicente teaches: Further comprising: converting the one or more facial landmarks to one or more two-dimensional and three-dimensional points (see Vicente at least [0031] two-dimensional facial feature points 207-210 and the projection of corresponding the 3D points);
translating the one or more two-dimensional and three-dimensional points to one or more three-dimensional vectors (see Vicente at least [0036] 3D translational vector of the vehicle operator's head); and
converting the one or more three-dimensional vectors to a rotational matrix (see Vicente at least [0035] R is the rotation matrix defined by the head pose angles and [0057] determining the gaze vector v.sub.gaze based on the head pose angles).
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the movable device controlling method disclosed by Deyaf, Kohlhoff, Singh, and Wang to include the various facial feature pose parameter considerations of Vicente. One of ordinary skill in the art would have been motivated to make this modification because in order to more accurately track facial features of a user, the facial feature parameters may be used to minimize fitting error related to the facial tracking, as suggested by Vicente (see Vicente at least [0038] The overall fitting error E, which is the total fitting error of all the facial feature points 207-210, is minimized with respect to the pose parameters (i.e., the rotation matrix R, the 3D translation vector t.sub.head.sub.P, scale factor s and shape coefficient β using an alternating optimization approach).
Regarding claim 12, Deyaf, Kohlhoff, Singh, and Wang disclose: The system of claim 10.
Deyaf, Kohlhoff, Singh, and Wang do not teach: wherein the instructions further cause the one or more processors to: convert the one or more facial landmarks to one or more two-dimensional and three-dimensional points; translate the one or more two-dimensional and three-dimensional points to one or more three-dimensional vectors; and convert the one or more three-dimensional vectors to a rotational matrix.
However, Vicente teaches: wherein the instructions further cause the one or more processors to: convert the one or more facial landmarks to one or more two-dimensional and three-dimensional points (see Vicente at least [0031] two-dimensional facial feature points 207-210 and the projection of corresponding the 3D points);
translate the one or more two-dimensional and three-dimensional points to one or more three-dimensional vectors (see Vicente at least [0036] 3D translational vector of the vehicle operator's head); and
convert the one or more three-dimensional vectors to a rotational matrix (see Vicente at least [0035] R is the rotation matrix defined by the head pose angles and [0057] determining the gaze vector v.sub.gaze based on the head pose angles).
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the movable device controlling system disclosed by Deyaf, Kohlhoff, Singh, and Wang to include the various facial feature pose parameter considerations of Vicente. One of ordinary skill in the art would have been motivated to make this modification because in order to more accurately track facial features of a user, the facial feature parameters may be used to minimize fitting error related to the facial tracking, as suggested by Vicente (see Vicente at least [0038] The overall fitting error E, which is the total fitting error of all the facial feature points 207-210, is minimized with respect to the pose parameters (i.e., the rotation matrix R, the 3D translation vector t.sub.head.sub.P, scale factor s and shape coefficient β using an alternating optimization approach).
Regarding claim 20, Deyaf, Kohlhoff, Singh, and Wang disclose: The non-transitory computer-readable storage medium of claim 17.
Deyaf, Kohlhoff, Singh, and Wang do not teach: wherein the instructions further cause the one or more processors to: convert the one or more facial landmarks to one or more two-and-three-dimensional points; translate the one or more two-and-three-dimensional points to one or more three- dimensional vectors; and convert the one or more three-dimensional vectors to a rotational matrix.
However, Vicente teaches: wherein the instructions further cause the one or more processors to: convert the one or more facial landmarks to one or more two-and-three-dimensional points (see Vicente at least [0031] two-dimensional facial feature points 207-210 and the projection of corresponding the 3D points);
translate the one or more two-and-three-dimensional points to one or more three- dimensional vectors (see Vicente at least [0036] 3D translational vector of the vehicle operator's head); and
convert the one or more three-dimensional vectors to a rotational matrix (see Vicente at least [0035] R is the rotation matrix defined by the head pose angles and [0057] determining the gaze vector v.sub.gaze based on the head pose angles).
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the movable device controlling computer-readable medium disclosed by Deyaf, Kohlhoff, Singh, and Wang to include the various facial feature pose parameter considerations of Vicente. One of ordinary skill in the art would have been motivated to make this modification because in order to more accurately track facial features of a user, the facial feature parameters may be used to minimize fitting error related to the facial tracking, as suggested by Vicente (see Vicente at least [0038] The overall fitting error E, which is the total fitting error of all the facial feature points 207-210, is minimized with respect to the pose parameters (i.e., the rotation matrix R, the 3D translation vector t.sub.head.sub.P, scale factor s and shape coefficient β using an alternating optimization approach).
Claim(s) 6, 14, and 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Deyaf