Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 12/03/2025 has been entered.
Response to Arguments
Applicant’s arguments filed 12/03/2025, with respect to the rejections of independent claims 1, 9, and 16 and their respective dependent claims under 35 USC 103 have been fully considered and are persuasive. Therefore, the rejections have been withdrawn. However, upon further consideration, new grounds of rejection are made in view of the previously applied references from Oh, Parshionikar, and Paul in addition to a newly applied reference from Gupta et al. (US 20210312024 A1), hereinafter Gupta. Specifically, Gupta teaches the claimed scenario in which multiple sets of facial features are detected during some unlock process, with different sets of facial features corresponding to different applications to be surfaced/activated, as presented in at least the amended limitations of Claim 1: “during the unlock signal detection process of the user device,(i) the set of unlock facial features capturing the second distance between the first facial feature and the second facial feature of the user is detected and (ii) the set of intent facial features capturing the first distance between the first facial feature and the second facial feature of the user is not detected”.
Claim Objections
Claim 16 is objected to because of the following informalities:
In line 23-25 of Claim 16, the limitation “the second set of facial features capturing the first distance between the first facial feature and the second facial feature of the user” should read: “the second set of facial features capturing the second distance between the first facial feature and the second facial feature of the user”
Appropriate correction is required.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3, 8-12, 15, 18, and 21 are rejected under 35 U.S.C. 103 as being unpatentable over Oh et al. (US 20140313307 A1), hereinafter Oh, in view of Parshionikar (US 20210157402 A1), hereinafter Parshionikar, Paul et al. (US 20220237274 A1), hereinafter Paul, and Gupta et al. (US 20210312024 A1), hereinafter Gupta.
Regarding Claim 1:
Oh teaches a system, comprising: one or more processors (Oh – Paragraph [0027]: Referring to FIG. 1, the device 100 includes a touch screen 190 and a touch screen controller 195. Also, the device 100 includes a controller 110; and Paragraph [0028]: The controller 110 may include a CPU 111); one or more hardware storage devices storing instructions that are executable by the one or more processors to configure the system to (Oh – Paragraph [0028]: The controller 110 may include a CPU 111, a ROM 112 that stores a control program for controlling the device 100, and a RAM 113 that stores a signal or data input from the outside of the device 100 or is used as a memory region for an operation performed in the device 100. The CPU 111 may include a single-core, a dual-core, a triple-core, or a quad-core processor; and Paragraph [0099]: The memory may be an example of machine-readable storage media that are suitable for storing a program including instructions to implement the embodiments, or programs. Therefore, the invention may include a program including a code to implement an apparatus or a method as claimed herein, and a machine-readable storage medium including the program, for example, a computer-readable storage medium); detect a set of facial features of a user using one or more sensors of a user device, the set of facial features of the user comprising a set of unlock features of the user and a set of intent facial features of the user (Oh – Paragraph [0027]: Referring to FIG. 1, the device 100 includes a touch screen 190 and a touch screen controller 195. Also, the device 100 includes a controller 110, the mobile communication module 120, the sub-communication module 130, a multimedia module 140, a camera module 150, a GPS module 155, an input/output module 160, a sensor module 170, a storage unit 175, and a power supply unit 180; and Paragraph [0060]: The electronic device 100 extracts a facial region in operation 206 when the request for canceling the lock mode is input. That is, the electronic device 100 captures an image using one of the first camera 151 and the second camera 152, and extracts the facial region from the captured image; and Paragraph [0061]: When the facial region is extracted, the electronic device 100 proceeds with a facial recognition process 210 corresponding to steps 212 through 216, and proceeds with a gaze recognition process 220 corresponding to steps 222 through 226; [0062]: In the facial recognition process 210, the electronic device 100 extracts component information associated with the facial region in step 212. For example, the electronic device 100 extracts, from the facial region, facial component information such as a symmetric composition, a shape, hair, a color of eyes, muscles of a face, and the like; [0063] In step 214, the electronic device 100 compares the extracted facial component information to a face registered in advance. In this scenario, the registered face may be facial component information associated with a facial region stored in advance by the user after capturing a user's face; [0064] In step 216, the electronic device 100 determines whether the facial region is identical to the registered face. When the facial region is identical to the registered face, the electronic device 100 proceeds to step 230; [0065] In the gaze recognition process 220, the electronic device 100 extracts an eye region from the extracted facial region in step 222. In step 224, the electronic device 100 recognizes a gaze by analyzing a pupil and a reflected light in the extracted eye region, and determines a point at which the recognized gaze is aimed. In this example, the electronic device 100 may display a gaze image 312 to represent the recognized gaze, as shown in the FIG. 3A; [0067] After performing the facial recognition process 210 and the gaze recognition process 220 as described above, the electronic device 100 performs unlocking based on a result of the facial recognition process 210 indicating that faces are identical, and simultaneously, executes the function selected as a result of the gaze recognition process 220); bind the set of intent facial features with a particular application of a plurality of applications of the user device, wherein binding the set of intent facial features with the particular application causes the particular application to be surfaced for display on the user device upon unlocking of the user device when the set of facial features is detected during an unlock signal detection process of the user device (Oh – Paragraph [0066]: The electronic device 100 temporarily stores a function placed on the point at which the gaze is aimed in step 226. For example, when the recognized gaze is aimed at the Internet browser icon 314-4 as shown in the FIG. 3A, the electronic device 100 selects an Internet browser function, and temporarily stores the selected function. The electronic device 100 may select a function placed on the point at which the recognized gaze is aimed at, and temporarily stores the selected function, and proceeds to step 230; [0067]: After performing the facial recognition process 210 and the gaze recognition process 220 as described above, the electronic device 100 performs unlocking based on a result of the facial recognition process 210 indicating that faces are identical, and simultaneously, executes the function selected as a result of the gaze recognition process 220; [0068]: For example, when the facial recognition shows that the recognized face is identical to the registered face and an internet browser function (or application) is selected through the gaze recognition, the electronic device 100 performs unlocking and simultaneously executes the Internet browser function, and displays an Internet browser screen 320, as shown in the FIG. 3B); [configure the user device to cause content that was displayed on the user device when the user device entered a locked state to be surfaced for display on the user device] upon unlocking of the user device when the set of unlock facial features is detected and the set of intent facial features is not detected during the unlock signal detection process of the user device (Oh – Paragraph [0059]: The electronic device 100 determines whether a request for unlocking exists in step 204; [0062]: In the facial recognition process 210, the electronic device 100 extracts component information associated with the facial region in step 212. For example, the electronic device 100 extracts, from the facial region, facial component information such as a symmetric composition, a shape, hair, a color of eyes, muscles of a face, and the like; [0067]: After performing the facial recognition process 210 and the gaze recognition process 220 as described above, the electronic device 100 performs unlocking based on a result of the facial recognition process 210 indicating that faces are identical, and simultaneously, executes the function selected as a result of the gaze recognition process 220; [0096]: Alternatively, if a function is not selected due to the failure of the gaze recognition and the like, only unlocking may be performed).
Oh does not expressly teach wherein the set of intent facial features captures a first distance between a first facial feature and a second facial feature of the user, and wherein the set of unlock facial features captures a second distance between the first facial feature and the second facial feature of the user, wherein the second distance is different from the first distance; the set of unlock facial features capturing the second distance between the first facial feature and the second facial feature of the user; the set of intent facial features capturing the first distance between the first facial feature and the second facial feature of the user.
However, Parshionikar teaches wherein the set of intent facial features captures a first distance between a first facial feature and a second facial feature of the user, and wherein the set of unlock facial features captures a second distance between the first facial feature and the second facial feature of the user, wherein the second distance is different from the first distance (Parshionikar – Paragraph [0137]: For example, to detect the facial expression of a smile, the mouth can be considered to be a key feature and various points of interest on the mouth can be tracked in relation to each other as well as to the positions they were in during the calibration/initialization process. The change in position of corners of mouth relative to each other and/or center of the mouth can provide an indication of level of smile being expressed by the user. Typically, the mouth corners move away from each other when a user smiles. Such changes in position of the corners can be used to determine the level of smile or other facial expressions involving the mouth. As an example, if the distance between two corners of mouth during calibration/initialization was d1, whereas the distance between the two corner changes to d2 during a facial expression involving the mouth, then magnitude (level) of that expression can be calculated as following. Magnitude=(d2−d1)*100/d1; and Paragraph [0138]: Many other such formulae based on combination of location of points of interest on the user's face (such corners of mouth, corners of eyes, mid points of eye lids, center of pupil of the eye, center of the chin, center of upper/lower lip, tip of the nose, nostril, start/mid/end of eye brows, etc.) can be utilized. The relative locations (distance) between various points of interest and the change in those distances when going from one point in time to another can be utilized to derive a numerical value of the magnitude of a facial expression; and Paragraph [0010]: This application includes disclosure of methods, systems, apparatuses as well as principles/algorithms that can be implemented using computer executable instructions stored on computer readable mediums, for defining user gestures, performing user gestures, interpreting user actions, detecting user intent, confirming user intent and communicating user intent when communicating with electronic devices; and Paragraph [0519]: Some electronic devices use facial recognition for securing access to devices. For example, Apple's iPhone X allows unlocking a locked phone by means of the facial recognition that will unlock only when the face of the user looks similar to the face of the user authorized to use that phone. However, this arrangement can get fooled by having an unauthorized user use a mask that resembles the authorized user. Further, a twin sibling or even a relative of the authorized user can gain access to the phone due to the resemblance of their face with the authorized user. Such systems can be made more fool proof by requiring the user to present additional actions to unlock the device) the set of unlock facial features capturing the second distance between the first facial feature and the second facial feature of the user (Parshionikar – Paragraph [0137]: For example, to detect the facial expression of a smile, the mouth can be considered to be a key feature and various points of interest on the mouth can be tracked in relation to each other as well as to the positions they were in during the calibration/initialization process. The change in position of corners of mouth relative to each other and/or center of the mouth can provide an indication of level of smile being expressed by the user. Typically, the mouth corners move away from each other when a user smiles. Such changes in position of the corners can be used to determine the level of smile or other facial expressions involving the mouth. As an example, if the distance between two corners of mouth during calibration/initialization was d1, whereas the distance between the two corner changes to d2 during a facial expression involving the mouth, then magnitude (level) of that expression can be calculated as following. Magnitude=(d2−d1)*100/d1; and Paragraph [0138]: Many other such formulae based on combination of location of points of interest on the user's face (such corners of mouth, corners of eyes, mid points of eye lids, center of pupil of the eye, center of the chin, center of upper/lower lip, tip of the nose, nostril, start/mid/end of eye brows, etc.) can be utilized. The relative locations (distance) between various points of interest and the change in those distances when going from one point in time to another can be utilized to derive a numerical value of the magnitude of a facial expression; and Paragraph [0010]: This application includes disclosure of methods, systems, apparatuses as well as principles/algorithms that can be implemented using computer executable instructions stored on computer readable mediums, for defining user gestures, performing user gestures, interpreting user actions, detecting user intent, confirming user intent and communicating user intent when communicating with electronic devices; and Paragraph [0519]: Some electronic devices use facial recognition for securing access to devices. For example, Apple's iPhone X allows unlocking a locked phone by means of the facial recognition that will unlock only when the face of the user looks similar to the face of the user authorized to use that phone. However, this arrangement can get fooled by having an unauthorized user use a mask that resembles the authorized user. Further, a twin sibling or even a relative of the authorized user can gain access to the phone due to the resemblance of their face with the authorized user. Such systems can be made more fool proof by requiring the user to present additional actions to unlock the device); the set of intent facial features capturing the first distance between the first facial feature and the second facial feature of the user (Parshionikar – Paragraph [0137]: For example, to detect the facial expression of a smile, the mouth can be considered to be a key feature and various points of interest on the mouth can be tracked in relation to each other as well as to the positions they were in during the calibration/initialization process. The change in position of corners of mouth relative to each other and/or center of the mouth can provide an indication of level of smile being expressed by the user. Typically, the mouth corners move away from each other when a user smiles. Such changes in position of the corners can be used to determine the level of smile or other facial expressions involving the mouth. As an example, if the distance between two corners of mouth during calibration/initialization was d1, whereas the distance between the two corner changes to d2 during a facial expression involving the mouth, then magnitude (level) of that expression can be calculated as following. Magnitude=(d2−d1)*100/d1; and Paragraph [0138]: Many other such formulae based on combination of location of points of interest on the user's face (such corners of mouth, corners of eyes, mid points of eye lids, center of pupil of the eye, center of the chin, center of upper/lower lip, tip of the nose, nostril, start/mid/end of eye brows, etc.) can be utilized. The relative locations (distance) between various points of interest and the change in those distances when going from one point in time to another can be utilized to derive a numerical value of the magnitude of a facial expression; and Paragraph [0010]: This application includes disclosure of methods, systems, apparatuses as well as principles/algorithms that can be implemented using computer executable instructions stored on computer readable mediums, for defining user gestures, performing user gestures, interpreting user actions, detecting user intent, confirming user intent and communicating user intent when communicating with electronic devices; and Paragraph [0519]: Some electronic devices use facial recognition for securing access to devices. For example, Apple's iPhone X allows unlocking a locked phone by means of the facial recognition that will unlock only when the face of the user looks similar to the face of the user authorized to use that phone. However, this arrangement can get fooled by having an unauthorized user use a mask that resembles the authorized user. Further, a twin sibling or even a relative of the authorized user can gain access to the phone due to the resemblance of their face with the authorized user. Such systems can be made more fool proof by requiring the user to present additional actions to unlock the device).
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to modify Oh, further incorporating Parshionikar to arrive at the conclusion of the claimed invention. One would be motivated to incorporate Parshionikar’s analysis of at least two facial images to evaluate changes in distances between at least two facial points of interest into Oh’s system for collecting and associating user facial features/gestures with device functions. This combination would enhance Oh’s system by providing the capability to more precisely analyze user facial expressions and changes thereof in determining gesture-based user intent.
The combination of Oh and Parshionikar does not expressly teach cause content that was displayed on the user device when the user device entered a locked state to be surfaced for display on the user device upon unlocking of the user device.
However, Paul teaches and configure the user device to cause content that was displayed on the user device when the user device entered a locked state to be surfaced for display on the user device upon unlocking of the user device (Paul – Paragraph [0299]: At FIG. 7T, because accessory-based unlocking criteria have been met, computer system 700 transitions from the locked state to an unlocked state. Because accessory-based unlocking criteria have been met, computer system 700 also replaces lock indicator 712a with unlock indicator 712b on display 710, as illustrated in FIGS. 7S-7T. In some embodiments, after displaying the user interface of FIG. 7T, computer system 700 can display one or more user interfaces that would have been previously restricted to the user if authentication were not successful, such as a screen with multiple application icons (e.g., as shown and described below in FIG. 7W) and/or a user interface that was previously displayed before computer system 700 was transitioned from the unlocked state to the locked).
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to modify Oh and Parshionikar, further incorporating Paul to arrive at the conclusion of the claimed invention. One would be motivated to incorporate Paul’s teaching to restore the device to its state prior to locking unless otherwise specified by some user action into Oh and Parshionikar’s combined system for collecting and associating user facial features/gestures with device functions. This combined functionality would enhance the efficiency and convenience of the device and biometric authentication thereof.
The combination of Oh, Parshionikar, and Paul does not expressly teach when, during the unlock signal detection process of the user device,(i) the set of unlock facial features … is detected and (ii) the set of intent facial features … is not detected.
However, Gupta teaches when, during the unlock signal detection process of the user device (Gupta – Paragraph [0079]: The combination of facial recognition and mien detection can be utilized in a variety of ways. For example, in one or more embodiments when the at least one image sufficiently corresponds to the one or more predefined reference images 108 (facial recognition) and the one or more images fail to comprise the depiction of the predefined mien expressed by the user 101 (no mien detection), the one or more processors 110 grant only limited operational access to features, applications, or data of the electronic device. However, when the at least one image sufficiently corresponds to the one or more predefined reference images 108 (facial recognition) and the one or more images comprise the depiction of the predefined mien expressed by the user (mien detection), the one or more processors 110 grant full operational access to the features, the applications, or the data of the electronic device in one or more embodiments; and Paragraph [0083]: This two-step process allows the user 101 to be facially authenticated at step 116 before expressing the mien to grant additional access at step 121),(i) the set of unlock facial features … is detected (Gupta – Paragraph [0051]: In FIG. 1, a user 101 is initially authenticating himself as an authorized user of the electronic device 100 via facial recognition to gain limited operational access to features, services, applications, data, content, or other properties of the electronic device 100; and Paragraph [0057]: For example, the limited operational access may unlock the electronic device 100, thereby allowing the user 101 to use the operating system and some features of the electronic device 100) and (ii) the set of intent facial features … is not detected (Gupta – Paragraph [0047]: As noted above, in one or more embodiments each folder, application, data, or feature of the electronic device can be locked with its own predefined mien. To access the financial application, for example, the authorized user may have to express a first mien, such as closing one eye. To access the health application, by contrast, the authorized user may have to express a second mien, such as sticking out their tongue, and so forth; and Paragraph [0079]: The combination of facial recognition and mien detection can be utilized in a variety of ways. For example, in one or more embodiments when the at least one image sufficiently corresponds to the one or more predefined reference images 108 (facial recognition) and the one or more images fail to comprise the depiction of the predefined mien expressed by the user 101 (no mien detection), the one or more processors 110 grant only limited operational access to features, applications, or data of the electronic device; and Paragraph [0057]: For example, the limited operational access may unlock the electronic device 100; and Paragraph [0083]: This two-step process allows the user 101 to be facially authenticated at step 116 before expressing the mien to grant additional access at step 121).
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to modify Oh, Parshionikar, and Paul, further incorporating Gupta to arrive at the conclusion of the claimed invention. One would be motivated to incorporate Gupta’s process for optionally detecting multiple user facial expressions corresponding to multiple device functions including initial unlocking and further application selection into Oh, Parshionikar, and Paul’s combined system for collecting and associating user facial features/gestures with device functions. This addition would enhance the security of the system by establishing a sort of MFA process incorporating user facial features to access particular applications, as well as adding convenience to the user experience.
Regarding Claim 2:
The combination of Oh, Parshionikar, Paul, and Gupta teaches the system of claim 1.
Paul further teaches wherein the unlocking of the user device comprises transitioning the user device from the locked state to an unlocked state (Paul – Paragraph [0299]: At FIG. 7T, because accessory-based unlocking criteria have been met, computer system 700 transitions from the locked state to an unlocked state. Because accessory-based unlocking criteria have been met, computer system 700 also replaces lock indicator 712a with unlock indicator 712b on display 710, as illustrated in FIGS. 75-7T).
The motivation to combine the arts is the same as that of Claim 1.
Regarding Claim 3:
The combination of Oh, Parshionikar, Paul, and Gupta teaches the system of claim 2.
Oh further teaches wherein binding the set of intent facial features with the particular application causes the particular application, [rather than the content that was displayed on the user device when the user device entered the locked state,] to be surfaced for display on the user device upon unlocking of the user device when the set of facial features is detected during the unlock signal detection process of the user device (Oh – Paragraph [0066]: The electronic device 100 temporarily stores a function placed on the point at which the gaze is aimed in step 226. For example, when the recognized gaze is aimed at the Internet browser icon 314-4 as shown in the FIG. 3A, the electronic device 100 selects an Internet browser function, and temporarily stores the selected function. The electronic device 100 may select a function placed on the point at which the recognized gaze is aimed at, and temporarily stores the selected function, and proceeds to step 230; [0067]: After performing the facial recognition process 210 and the gaze recognition process 220 as described above, the electronic device 100 performs unlocking based on a result of the facial recognition process 210 indicating that faces are identical, and simultaneously, executes the function selected as a result of the gaze recognition process 220; [0068]: For example, when the facial recognition shows that the recognized face is identical to the registered face and an internet browser function (or application) is selected through the gaze recognition, the electronic device 100 performs unlocking and simultaneously executes the Internet browser function, and displays an Internet browser screen 320, as shown in the FIG. 3B).
Paul further teaches rather than the content that was displayed on the user device when the user device entered the locked state (Paul – Paragraph [0299]: At FIG. 7T, because accessory-based unlocking criteria have been met, computer system 700 transitions from the locked state to an unlocked state. Because accessory-based unlocking criteria have been met, computer system 700 also replaces lock indicator 712a with unlock indicator 712b on display 710, as illustrated in FIGS. 7S-7T. In some embodiments, after displaying the user interface of FIG. 7T, computer system 700 can display one or more user interfaces that would have been previously restricted to the user if authentication were not successful, such as a screen with multiple application icons (e.g., as shown and described below in FIG. 7W) and/or a user interface that was previously displayed before computer system 700 was transitioned from the unlocked state to the locked).
The motivation to combine the arts is the same as that of Claim 1.
Regarding Claim 8:
The combination of Oh, Parshionikar, Paul, and Gupta teaches the system of claim 1.
Oh further teaches wherein the system comprises the user device (Oh – Paragraph [0009]: Accordingly, aspects of the present invention provide an electronic device and an unlocking method in the electronic device which simultaneously and promptly requests execution of a desired function or application before a user cancels a lock mode, and executes the function or application requested by the user at the same time as canceling the lock mode; [0055]: The user input may include various forms of information that are input into the device 100, such as, for example, a gesture of the user, a voice, a movement of an eye, a bio-signal, and the like).
The motivation to combine the arts is the same as that of Claim 1.
Regarding Claim 9:
Oh teaches a system, comprising: one or more processors (Oh – Paragraph [0027]: Referring to FIG. 1, the device 100 includes a touch screen 190 and a touch screen controller 195. Also, the device 100 includes a controller 110; [0028]: The controller 110 may include a CPU 111); one or more hardware storage devices storing instructions that are executable by the one or more processors to configure the system to (Oh – Paragraph [0028]: The controller 110 may include a CPU 111, a ROM 112 that stores a control program for controlling the device 100, and a RAM 113 that stores a signal or data input from the outside of the device 100 or is used as a memory region for an operation performed in the device 100. The CPU 111 may include a single-core, a dual-core, a triple-core, or a quad-core processor; and Paragraph [0099]: The memory may be an example of machine-readable storage media that are suitable for storing a program including instructions to implement the embodiments, or programs. Therefore, the invention may include a program including a code to implement an apparatus or a method as claimed herein, and a machine-readable storage medium including the program, for example, a computer-readable storage medium); detect a set of facial features of a user using one or more sensors of a user device during an unlock signal detection process of the user device (Oh – Paragraph [0027]: Referring to FIG. 1, the device 100 includes a touch screen 190 and a touch screen controller 195. Also, the device 100 includes a controller 110, the mobile communication module 120, the sub-communication module 130, a multimedia module 140, a camera module 150, a GPS module 155, an input/output module 160, a sensor module 170, a storage unit 175, and a power supply unit 180; [0060]: The electronic device 100 extracts a facial region in operation 206 when the request for canceling the lock mode is input. That is, the electronic device 100 captures an image using one of the first camera 151 and the second camera 152, and extracts the facial region from the captured image; [0061]: When the facial region is extracted, the electronic device 100 proceeds with a facial recognition process 210 corresponding to steps 212 through 216, and proceeds with a gaze recognition process 220 corresponding to steps 222 through 226; [0062]: In the facial recognition process 210, the electronic device 100 extracts component information associated with the facial region in step 212. For example, the electronic device 100 extracts, from the facial region, facial component information such as a symmetric composition, a shape, hair, a color of eyes, muscles of a face, and the like; [0063]: In step 214, the electronic device 100 compares the extracted facial component information to a face registered in advance. In this scenario, the registered face may be facial component information associated with a facial region stored in advance by the user after capturing a user's face; [0064]: In step 216, the electronic device 100 determines whether the facial region is identical to the registered face. When the facial region is identical to the registered face, the electronic device 100 proceeds to step 230; [0065]: In the gaze recognition process 220, the electronic device 100 extracts an eye region from the extracted facial region in step 222. In step 224, the electronic device 100 recognizes a gaze by analyzing a pupil and a reflected light in the extracted eye region, and determines a point at which the recognized gaze is aimed. In this example, the electronic device 100 may display a gaze image 312 to represent the recognized gaze, as shown in the FIG. 3A; [0067]: After performing the facial recognition process 210 and the gaze recognition process 220 as described above, the electronic device 100 performs unlocking based on a result of the facial recognition process 210 indicating that faces are identical, and simultaneously, executes the function selected as a result of the gaze recognition process 220); when the set of facial features is determined to include a set of unlock facial features of the user and a set of intent facial features of the user, the set of intent facial features being bound to a particular application of a plurality of applications of the user device, surface the particular application for display on the user device upon unlocking of the user device (Oh – Paragraph [0060]: The electronic device 100 extracts a facial region in operation 206 when the request for canceling the lock mode is input. That is, the electronic device 100 captures an image using one of the first camera 151 and the second camera 152, and extracts the facial region from the captured image; [0061]: When the facial region is extracted, the electronic device 100 proceeds with a facial recognition process 210 corresponding to steps 212 through 216, and proceeds with a gaze recognition process 220 corresponding to steps 222 through 226; [0062]: In the facial recognition process 210, the electronic device 100 extracts component information associated with the facial region in step 212. For example, the electronic device 100 extracts, from the facial region, facial component information such as a symmetric composition, a shape, hair, a color of eyes, muscles of a face, and the like; [0063]: In step 214, the electronic device 100 compares the extracted facial component information to a face registered in advance. In this scenario, the registered face may be facial component information associated with a facial region stored in advance by the user after capturing a user's face; [0064]: In step 216, the electronic device 100 determines whether the facial region is identical to the registered face. When the facial region is identical to the registered face, the electronic device 100 proceeds to step 230; [0065]: In the gaze recognition process 220, the electronic device 100 extracts an eye region from the extracted facial region in step 222. In step 224, the electronic device 100 recognizes a gaze by analyzing a pupil and a reflected light in the extracted eye region, and determines a point at which the recognized gaze is aimed. In this example, the electronic device 100 may display a gaze image 312 to represent the recognized gaze, as shown in the FIG. 3A; [0067]: After performing the facial recognition process 210 and the gaze recognition process 220 as described above, the electronic device 100 performs unlocking based on a result of the facial recognition process 210 indicating that faces are identical, and simultaneously, executes the function selected as a result of the gaze recognition process 220; [0066]: The electronic device 100 temporarily stores a function placed on the point at which the gaze is aimed in step 226. For example, when the recognized gaze is aimed at the Internet browser icon 314-4 as shown in the FIG. 3A, the electronic device 100 selects an Internet browser function, and temporarily stores the selected function. The electronic device 100 may select a function placed on the point at which the recognized gaze is aimed at, and temporarily stores the selected function, and proceeds to step 230; [0067]: After performing the facial recognition process 210 and the gaze recognition process 220 as described above, the electronic device 100 performs unlocking based on a result of the facial recognition process 210 indicating that faces are identical, and simultaneously, executes the function selected as a result of the gaze recognition process 220; [0068]: For example, when the facial recognition shows that the recognized face is identical to the registered face and an internet browser function (or application) is selected through the gaze recognition, the electronic device 100 performs unlocking and simultaneously executes the Internet browser function, and displays an Internet browser screen 320, as shown in the FIG. 3B); and when the set of facial features is determined to include the set of unlock facial features and not include the set of intent facial features, [surface content that was displayed on the user device when the user device entered a locked state] (Oh – Paragraph [0059]: The electronic device 100 determines whether a request for unlocking exists in step 204; [0062]: In the facial recognition process 210, the electronic device 100 extracts component information associated with the facial region in step 212. For example, the electronic device 100 extracts, from the facial region, facial component information such as a symmetric composition, a shape, hair, a color of eyes, muscles of a face, and the like; [0067]: After performing the facial recognition process 210 and the gaze recognition process 220 as described above, the electronic device 100 performs unlocking based on a result of the facial recognition process 210 indicating that faces are identical, and simultaneously, executes the function selected as a result of the gaze recognition process 220; [0096]: Alternatively, if a function is not selected due to the failure of the gaze recognition and the like, only unlocking may be performed).
Oh does not expressly teach wherein the set of intent facial features captures a first distance between a first facial feature and a second facial feature of the user, and wherein the set of unlock facial features captures a second distance between the first facial feature and the second facial feature of the user, wherein the second distance is different from the first distance; the set of unlock facial features capturing the second distance between the first facial feature and the second facial feature of the user; the set of intent facial features capturing the first distance between the first facial feature and the second facial feature of the user.
However, Parshionikar teaches wherein the set of intent facial features captures a first distance between a first facial feature and a second facial feature of the user, and wherein the set of unlock facial features captures a second distance between the first facial feature and the second facial feature of the user, wherein the second distance is different from the first distance (Parshionikar – Paragraph [0137]: For example, to detect the facial expression of a smile, the mouth can be considered to be a key feature and various points of interest on the mouth can be tracked in relation to each other as well as to the positions they were in during the calibration/initialization process. The change in position of corners of mouth relative to each other and/or center of the mouth can provide an indication of level of smile being expressed by the user. Typically, the mouth corners move away from each other when a user smiles. Such changes in position of the corners can be used to determine the level of smile or other facial expressions involving the mouth. As an example, if the distance between two corners of mouth during calibration/initialization was d1, whereas the distance between the two corner changes to d2 during a facial expression involving the mouth, then magnitude (level) of that expression can be calculated as following. Magnitude=(d2−d1)*100/d1; and Paragraph [0138]: Many other such formulae based on combination of location of points of interest on the user's face (such corners of mouth, corners of eyes, mid points of eye lids, center of pupil of the eye, center of the chin, center of upper/lower lip, tip of the nose, nostril, start/mid/end of eye brows, etc.) can be utilized. The relative locations (distance) between various points of interest and the change in those distances when going from one point in time to another can be utilized to derive a numerical value of the magnitude of a facial expression; and Paragraph [0010]: This application includes disclosure of methods, systems, apparatuses as well as principles/algorithms that can be implemented using computer executable instructions stored on computer readable mediums, for defining user gestures, performing user gestures, interpreting user actions, detecting user intent, confirming user intent and communicating user intent when communicating with electronic devices; and Paragraph [0519]: Some electronic devices use facial recognition for securing access to devices. For example, Apple's iPhone X allows unlocking a locked phone by means of the facial recognition that will unlock only when the face of the user looks similar to the face of the user authorized to use that phone. However, this arrangement can get fooled by having an unauthorized user use a mask that resembles the authorized user. Further, a twin sibling or even a relative of the authorized user can gain access to the phone due to the resemblance of their face with the authorized user. Such systems can be made more fool proof by requiring the user to present additional actions to unlock the device); the set of unlock facial features capturing the second distance between the first facial feature and the second facial feature of the user (Parshionikar – Paragraph [0137]: For example, to detect the facial expression of a smile, the mouth can be considered to be a key feature and various points of interest on the mouth can be tracked in relation to each other as well as to the positions they were in during the calibration/initialization process. The change in position of corners of mouth relative to each other and/or center of the mouth can provide an indication of level of smile being expressed by the user. Typically, the mouth corners move away from each other when a user smiles. Such changes in position of the corners can be used to determine the level of smile or other facial expressions involving the mouth. As an example, if the distance between two corners of mouth during calibration/initialization was d1, whereas the distance between the two corner changes to d2 during a facial expression involving the mouth, then magnitude (level) of that expression can be calculated as following. Magnitude=(d2−d1)*100/d1; and Paragraph [0138]: Many other such formulae based on combination of location of points of interest on the user's face (such corners of mouth, corners of eyes, mid points of eye lids, center of pupil of the eye, center of the chin, center of upper/lower lip, tip of the nose, nostril, start/mid/end of eye brows, etc.) can be utilized. The relative locations (distance) between various points of interest and the change in those distances when going from one point in time to another can be utilized to derive a numerical value of the magnitude of a facial expression; and Paragraph [0010]: This application includes disclosure of methods, systems, apparatuses as well as principles/algorithms that can be implemented using computer executable instructions stored on computer readable mediums, for defining user gestures, performing user gestures, interpreting user actions, detecting user intent, confirming user intent and communicating user intent when communicating with electronic devices; and Paragraph [0519]: Some electronic devices use facial recognition for securing access to devices. For example, Apple's iPhone X allows unlocking a locked phone by means of the facial recognition that will unlock only when the face of the user looks similar to the face of the user authorized to use that phone. However, this arrangement can get fooled by having an unauthorized user use a mask that resembles the authorized user. Further, a twin sibling or even a relative of the authorized user can gain access to the phone due to the resemblance of their face with the authorized user. Such systems can be made more fool proof by requiring the user to present additional actions to unlock the device); the set of intent facial features capturing the first distance between the first facial feature and the second facial feature of the user (Parshionikar – Paragraph [0137]: For example, to detect the facial expression of a smile, the mouth can be considered to be a key feature and various points of interest on the mouth can be tracked in relation to each other as well as to the positions they were in during the calibration/initialization process. The change in position of corners of mouth relative to each other and/or center of the mouth can provide an indication of level of smile being expressed by the user. Typically, the mouth corners move away from each other when a user smiles. Such changes in position of the corners can be used to determine the level of smile or other facial expressions involving the mouth. As an example, if the distance between two corners of mouth during calibration/initialization was d1, whereas the distance between the two corner changes to d2 during a facial expression involving the mouth, then magnitude (level) of that expression can be calculated as following. Magnitude=(d2−d1)*100/d1; and Paragraph [0138]: Many other such formulae based on combination of location of points of interest on the user's face (such corners of mouth, corners of eyes, mid points of eye lids, center of pupil of the eye, center of the chin, center of upper/lower lip, tip of the nose, nostril, start/mid/end of eye brows, etc.) can be utilized. The relative locations (distance) between various points of interest and the change in those distances when going from one point in time to another can be utilized to derive a numerical value of the magnitude of a facial expression; and Paragraph [0010]: This application includes disclosure of methods, systems, apparatuses as well as principles/algorithms that can be implemented using computer executable instructions stored on computer readable mediums, for defining user gestures, performing user gestures, interpreting user actions, detecting user intent, confirming user intent and communicating user intent when communicating with electronic devices; and Paragraph [0519]: Some electronic devices use facial recognition for securing access to devices. For example, Apple's iPhone X allows unlocking a locked phone by means of the facial recognition that will unlock only when the face of the user looks similar to the face of the user authorized to use that phone. However, this arrangement can get fooled by having an unauthorized user use a mask that resembles the authorized user. Further, a twin sibling or even a relative of the authorized user can gain access to the phone due to the resemblance of their face with the authorized user. Such systems can be made more fool proof by requiring the user to present additional actions to unlock the device).
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to modify Oh, further incorporating Parshionikar to arrive at the conclusion of the claimed invention. One would be motivated to incorporate Parshionikar’s analysis of at least two facial images to evaluate changes in distances between at least two facial points of interest into Oh’s system for collecting and associating user facial features/gestures with device functions. This combination would enhance Oh’s system by providing the capability to more precisely analyze user facial expressions and changes thereof in determining gesture-based user intent.
The combination of Oh and Parshionikar does not expressly teach surface content that was displayed on the user device when the user device entered a locked state.
However, Paul teaches surface content that was displayed on the user device when the user device entered a locked state (Paul – Paragraph [0299]: At FIG. 7T, because accessory-based unlocking criteria have been met, computer system 700 transitions from the locked state to an unlocked state. Because accessory-based unlocking criteria have been met, computer system 700 also replaces lock indicator 712a with unlock indicator 712b on display 710, as illustrated in FIGS. 7S-7T. In some embodiments, after displaying the user interface of FIG. 7T, computer system 700 can display one or more user interfaces that would have been previously restricted to the user if authentication were not successful, such as a screen with multiple application icons (e.g., as shown and described below in FIG. 7W) and/or a user interface that was previously displayed before computer system 700 was transitioned from the unlocked state to the locked).
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to modify Oh and Parshionikar, further incorporating Paul to arrive at the conclusion of the claimed invention. One would be motivated to incorporate Paul’s teaching to restore the device to its state prior to locking unless otherwise specified by some user action into Oh and Parshionikar’s combined system for collecting and associating user facial features/gestures with device functions. This combined functionality would enhance the efficiency and convenience of the device and biometric authentication thereof.
The combination of Oh, Parshionikar, and Paul does not expressly teach and when the set of facial features is determined to (i) include the set of unlock facial features … and (ii) not include the set of intent facial features.
However, Gupta teaches and when the set of facial features is determined to (i) include the set of unlock facial features … (Gupta – Paragraph [0051]: In FIG. 1, a user 101 is initially authenticating himself as an authorized user of the electronic device 100 via facial recognition to gain limited operational access to features, services, applications, data, content, or other properties of the electronic device 100; and Paragraph [0057]: For example, the limited operational access may unlock the electronic device 100, thereby allowing the user 101 to use the operating system and some features of the electronic device 100) and (ii) not include the set of intent facial features (Gupta – Paragraph [0047]: As noted above, in one or more embodiments each folder, application, data, or feature of the electronic device can be locked with its own predefined mien. To access the financial application, for example, the authorized user may have to express a first mien, such as closing one eye. To access the health application, by contrast, the authorized user may have to express a second mien, such as sticking out their tongue, and so forth; and Paragraph [0079]: The combination of facial recognition and mien detection can be utilized in a variety of ways. For example, in one or more embodiments when the at least one image sufficiently corresponds to the one or more predefined reference images 108 (facial recognition) and the one or more images fail to comprise the depiction of the predefined mien expressed by the user 101 (no mien detection), the one or more processors 110 grant only limited operational access to features, applications, or data of the electronic device; and Paragraph [0057]: For example, the limited operational access may unlock the electronic device 100; and Paragraph [0083]: This two-step process allows the user 101 to be facially authenticated at step 116 before expressing the mien to grant additional access at step 121).
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to modify Oh, Parshionikar, and Paul, further incorporating Gupta to arrive at the conclusion of the claimed invention. One would be motivated to incorporate Gupta’s process for optionally detecting multiple user facial expressions corresponding to multiple device functions including initial unlocking and further application selection into Oh, Parshionikar, and Paul’s combined system for collecting and associating user facial features/gestures with device functions. This addition would enhance the security of the system by establishing a sort of MFA process incorporating user facial features to access particular applications, as well as adding convenience to the user experience.
Regarding Claim 10:
The combination of Oh, Parshionikar, Paul, and Gupta teaches the system of claim 9.
Paul further teaches wherein the unlocking of the user device comprises transitioning the user device from the locked state to an unlocked state (Paul – Paragraph [0299]: At FIG. 7T, because accessory-based unlocking criteria have been met, computer system 700 transitions from the locked state to an unlocked state. Because accessory-based unlocking criteria have been met, computer system 700 also replaces lock indicator 712a with unlock indicator 712b on display 710, as illustrated in FIGS. 75-7T).
The motivation to combine the arts is the same as that of Claim 9.
Regarding Claim 11:
The combination of Oh, Parshionikar, Paul, and Gupta teaches the system of claim 10.
Oh further teaches wherein the particular application that becomes surfaced upon unlocking of the user device [is different from the content that was displayed on the user device when the user device entered the locked state prior to unlocking of the user device] (Oh – Paragraph [0066]: The electronic device 100 temporarily stores a function placed on the point at which the gaze is aimed in step 226. For example, when the recognized gaze is aimed at the Internet browser icon 314-4 as shown in the FIG. 3A, the electronic device 100 selects an Internet browser function, and temporarily stores the selected function. The electronic device 100 may select a function placed on the point at which the recognized gaze is aimed at, and temporarily stores the selected function, and proceeds to step 230; [0067]: After performing the facial recognition process 210 and the gaze recognition process 220 as described above, the electronic device 100 performs unlocking based on a result of the facial recognition process 210 indicating that faces are identical, and simultaneously, executes the function selected as a result of the gaze recognition process 220; [0068]: For example, when the facial recognition shows that the recognized face is identical to the registered face and an internet browser function (or application) is selected through the gaze recognition, the electronic device 100 performs unlocking and simultaneously executes the Internet browser function, and displays an Internet browser screen 320, as shown in the FIG. 3B).
Paul further teaches [wherein the particular application that becomes surfaced upon unlocking of the user device] is different from the content that was displayed on the user device when the user device entered the locked state prior to unlocking of the user device (Paul – Paragraph [0299]: At FIG. 7T, because accessory-based unlocking criteria have been met, computer system 700 transitions from the locked state to an unlocked state. Because accessory-based unlocking criteria have been met, computer system 700 also replaces lock indicator 712a with unlock indicator 712b on display 710, as illustrated in FIGS. 7S-7T. In some embodiments, after displaying the user interface of FIG. 7T, computer system 700 can display one or more user interfaces that would have been previously restricted to the user if authentication were not successful, such as a screen with multiple application icons (e.g., as shown and described below in FIG. 7W) and/or a user interface that was previously displayed before computer system 700 was transitioned from the unlocked state to the locked).
The motivation to combine the arts is the same as that of Claim 9.
Regarding Claim 12:
The combination of Oh, Parshionikar, Paul, and Gupta teaches the system of claim 9.
Oh further teaches wherein detecting the set of facial features comprises capturing a set of images using the one or more sensors of the user device (Oh – Paragraph[0027]: Referring to FIG. 1, the device 100 includes a touch screen 190 and a touch screen controller 195. Also, the device 100 includes a controller 110, the mobile communication module 120, the sub-communication module 130, a multimedia module 140, a camera module 150, a GPS module 155, an input/output module 160, a sensor module 170, a storage unit 175, and a power supply unit 180; [0060]: The electronic device 100 extracts a facial region in operation 206 when the request for canceling the lock mode is input. That is, the electronic device 100 captures an image using one of the first camera 151 and the second camera 152, and extracts the facial region from the captured image; [0061]: When the facial region is extracted, the electronic device 100 proceeds with a facial recognition process 210 corresponding to steps 212 through 216, and proceeds with a gaze recognition process 220 corresponding to steps 222 through 226; [0065]: In the gaze recognition process 220, the electronic device 100 extracts an eye region from the extracted facial region in step 222. In step 224, the electronic device 100 recognizes a gaze by analyzing a pupil and a reflected light in the extracted eye region, and determines a point at which the recognized gaze is aimed. In this example, the electronic device 100 may display a gaze image 312 to represent the recognized gaze, as shown in the FIG. 3A).
The motivation to combine the arts is the same as that of Claim 9.
Regarding Claim 15:
The combination of Oh, Parshionikar, Paul, and Gupta teaches the system of claim 9.
Oh further teaches wherein the system comprises the user device (Oh – Paragraph [0009]: Accordingly, aspects of the present invention provide an electronic device and an unlocking method in the electronic device which simultaneously and promptly requests execution of a desired function or application before a user cancels a lock mode, and executes the function or application requested by the user at the same time as canceling the lock mode; [0055]: The user input may include various forms of information that are input into the device 100, such as, for example, a gesture of the user, a voice, a movement of an eye, a bio-signal, and the like)
Regarding Claim 18:
The combination of Oh and Parshionikar teaches the system of claim 17.
Oh further teaches wherein binding the first set of facial features with the first application causes the first application, [rather than a different application that was displayed on the user device when the user device entered the locked state prior to the unlocking,] to be surfaced for display on the user device upon unlocking of the user device when the first set of facial features is detected during the unlock signal detection process of the user device (Oh – Paragraph [0066]: The electronic device 100 temporarily stores a function placed on the point at which the gaze is aimed in step 226. For example, when the recognized gaze is aimed at the Internet browser icon 314-4 as shown in the FIG. 3A, the electronic device 100 selects an Internet browser function, and temporarily stores the selected function. The electronic device 100 may select a function placed on the point at which the recognized gaze is aimed at, and temporarily stores the selected function, and proceeds to step 230; [0067]: After performing the facial recognition process 210 and the gaze recognition process 220 as described above, the electronic device 100 performs unlocking based on a result of the facial recognition process 210 indicating that faces are identical, and simultaneously, executes the function selected as a result of the gaze recognition process 220; [0068]: For example, when the facial recognition shows that the recognized face is identical to the registered face and an internet browser function (or application) is selected through the gaze recognition, the electronic device 100 performs unlocking and simultaneously executes the Internet browser function, and displays an Internet browser screen 320, as shown in the FIG. 3B).
The combination of Oh and Parshionikar does not expressly teach causes the first application, rather than a different application that was displayed on the user device when the user device entered the locked state prior to the unlocking, to be surfaced for display on the user device upon unlocking.
However, Paul teaches causes the first application, rather than a different application that was displayed on the user device when the user device entered the locked state prior to the unlocking, to be surfaced for display on the user device upon unlocking ... (Paul – Paragraph [0299]: At FIG. 7T, because accessory-based unlocking criteria have been met, computer system 700 transitions from the locked state to an unlocked state. Because accessory-based unlocking criteria have been met, computer system 700 also replaces lock indicator 712a with unlock indicator 712b on display 710, as illustrated in FIGS. 7S-7T. In some embodiments, after displaying the user interface of FIG. 7T, computer system 700 can display one or more user interfaces that would have been previously restricted to the user if authentication were not successful, such as a screen with multiple application icons (e.g., as shown and described below in FIG. 7W) and/or a user interface that was previously displayed before computer system 700 was transitioned from the unlocked state to the locked).
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to modify Oh and Parshionikar, further incorporating Paul to arrive at the conclusion of the claimed invention. One would be motivated to incorporate Paul’s teaching to specify the process of displaying desired content according to user intent into Oh and Parshionikar’s combined system for collecting and associating user facial features/gestures with device functions. This combined functionality would further improve ease of use and convenience for device users.
Regarding Claim 21:
The combination of Oh, Parshionikar, Paul, and Gupta teaches the system of claim 1.
Oh further teaches wherein the set of intent facial features comprises one or more facial landmarks, one or more facial nodal points, one or more face signatures, or one or more face tracking signals (Oh – Paragraph [0062]: In the facial recognition process 210, the electronic device 100 extracts, from the facial region, facial component information such as a symmetric composition, a shape, hair, a color of eyes, muscles of a face, and the like; [0065]: In the gaze recognition process 220, the electronic device 100 extracts an eye region from the extracted facial region in step 222. In step 224, the electronic device 100 recognizes a gaze by analyzing a pupil and a reflected light in the extracted eye region, and determines a point at which the recognized gaze is aimed; Examiner’s Comment: Examiner respectfully submits that the features extracted from both the facial recognition process and the gaze recognition process represent features that comprise at least one of facial landmarks and face tracking signals).
The motivation to combine the arts is the same as that of Claim 1.
Claim(s) 5-7 and 13-14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Oh in view of Parshionikar, Paul, Gupta, and Lin et al. (US 20190370530 A1), hereafter Lin.
Regarding Claim 5:
The combination of Oh, Parshionikar, Paul, and Gupta teaches the system of claim 1.
Oh further teaches wherein the set of facial features is extracted [from a set of image frames detected] using the one or more sensors, and wherein the set of unlock facial features is extracted [from a first subset of images of the set of image frames], and wherein the set of intent facial features is extracted [from a second subset of images of the set of image frames] (Oh – Paragraph [0027]: Referring to FIG. 1, the device 100 includes a touch screen 190 and a touch screen controller 195. Also, the device 100 includes a controller 110, the mobile communication module 120, the sub-communication module 130, a multimedia module 140, a camera module 150, a GPS module 155, an input/output module 160, a sensor module 170, a storage unit 175, and a power supply unit 180; [0060]: The electronic device 100 extracts a facial region in operation 206 when the request for canceling the lock mode is input. That is, the electronic device 100 captures an image using one of the first camera 151 and the second camera 152, and extracts the facial region from the captured image; [0061]: When the facial region is extracted, the electronic device 100 proceeds with a facial recognition process 210 corresponding to steps 212 through 216, and proceeds with a gaze recognition process 220 corresponding to steps 222 through 226; [0062]: In the facial recognition process 210, the electronic device 100 extracts component information associated with the facial region in step 212. For example, the electronic device 100 extracts, from the facial region, facial component information such as a symmetric composition, a shape, hair, a color of eyes, muscles of a face, and the like; [0063]: In step 214, the electronic device 100 compares the extracted facial component information to a face registered in advance. In this scenario, the registered face may be facial component information associated with a facial region stored in advance by the user after capturing a user's face; [0064]: In step 216, the electronic device 100 determines whether the facial region is identical to the registered face. When the facial region is identical to the registered face, the electronic device 100 proceeds to step 230; [0065]: In the gaze recognition process 220, the electronic device 100 extracts an eye region from the extracted facial region in step 222. In step 224, the electronic device 100 recognizes a gaze by analyzing a pupil and a reflected light in the extracted eye region, and determines a point at which the recognized gaze is aimed. In this example, the electronic device 100 may display a gaze image 312 to represent the recognized gaze, as shown in the FIG. 3A; [0067]: After performing the facial recognition process 210 and the gaze recognition process 220 as described above, the electronic device 100 performs unlocking based on a result of the facial recognition process 210 indicating that faces are identical, and simultaneously, executes the function selected as a result of the gaze recognition process 220).
Oh does not expressly teach wherein the set of facial features is extracted from a set of image frames detected using the one or more sensors, and wherein the set of unlock facial features is extracted from a first subset of images of the set of image frames, and wherein the set of intent facial features is extracted from a second subset of images of the set of image frames that is different from the first subset of images of the set of image frames.
However, Paul teaches wherein the set of facial features is extracted from a set of image frames detected using the one or more sensors, [and wherein the set of unlock facial features is extracted from a first subset of images of the set of image frames, and wherein the set of intent facial features is extracted from a second subset of images of the set of image frames that is different from the first subset of images of the set of image frames] (Paul – Paragraph [0101]: Device 100 optionally also includes one or more optical sensors 164. FIG. 1A shows an optical sensor coupled to optical sensor controller 158 in I/O subsystem 106. Optical sensor 164 optionally includes charge-coupled device (CCD) or complementary metal oxide semiconductor (CMOS) phototransistors. Optical sensor 164 receives light from the environment, projected through one or more lenses, and converts the light to data representing an image. In conjunction with imaging module 143 (also called a camera module), optical sensor 164 optionally captures still images or video; [0103]: In some embodiments, a depth map (e.g., depth map image) contains information (e.g., values) that relates to the distance of objects in a scene from a viewpoint (e.g., a camera, an optical sensor, a depth camera sensor). In one embodiment of a depth map, each depth pixel defines the position in the viewpoint's Z-axis where its corresponding two-dimensional pixel is located. In some embodiments, a depth map is composed of pixels wherein each pixel is defined by a value (e.g., 0-255). For example, the “0” value represents pixels that are located at the most distant place in a “three dimensional” scene and the “255” value represents pixels that are located closest to a viewpoint (e.g., a camera, an optical sensor, a depth camera sensor) in the “three dimensional” scene. In other embodiments, a depth map represents the distance between an object in a scene and the plane of the viewpoint. In some embodiments, the depth map includes information about the relative depth of various features of an object of interest in view of the depth camera (e.g., the relative depth of eyes, nose, mouth, ears of a user's face)).
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to modify Oh, further incorporating Paul to arrive at the conclusion of the claimed invention. One would be motivated to incorporate Paul’s teaching to collect facial features from a set of image frames captured by one or more sensors into Oh’s system for collecting and associating user facial features/gestures with device functions. This combined functionality would provide the system the potential to gather varying facial features corresponding to varying intentions.
The combination of Oh, Parshionikar, and Paul does not expressly teach wherein the set of facial features is extracted from a set of image frames detected using the one or more sensors, and wherein the set of unlock facial features is extracted from a first subset of images of the set of image frames, and wherein the set of intent facial features is extracted from a second subset of images of the set of image frames that is different from the first subset of images of the set of image frames.
However, Lin teaches wherein the set of facial features is extracted from a set of image frames detected [using the one or more sensors], and wherein the set of [unlock] facial features is extracted from a first subset of images of the set of image frames, and wherein the set of [intent] facial features is extracted from a second subset of images of the set of image frames that is different from the first subset of images of the set of image frames (Lin – Paragraph [0035]: A facial feature point is a point in an image that represents a facial feature, for example, a point that represents a facial feature or a face contour feature, and is usually represented in a form of coordinates. Samples can be obtained in various ways. For example, to ensure reliability of a sample set and improve accuracy of a preset error model, a plurality of pairs of adjacent frames of images may be extracted from a video including a face; [0034]: where each sample in the sample set includes facial feature points of a first image (or first reference image) that is a former one and facial feature points of the second image (or second reference image) that is a latter one in adjacent frames of images (or adjacent reference images); Examiner’s Comment: This teaching from Lin demonstrates a capability to gather facial features from different subsets of different sets of images captured sequentially by some optical sensor).
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to modify Oh, Parshionikar, Paul, and Gupta, further incorporating Lin to arrive at the conclusion of the claimed invention. One would be motivated to incorporate Lin’s teaching to analyze different parts of different image frames to determine verifiable facial features of a user into Oh, Parshionikar, Paul, and Gupta’s combined system for collecting and associating user facial features/gestures with device functions. This combined functionality would further enhance the system’s precision in gathering facial features/expressions to associate with user intent.
Regarding Claim 6:
The combination of Oh, Parshionikar, Paul, Gupta, and Lin teaches the system of claim 5.
Oh further teaches detect a second set of facial features of the user using the one or more sensors of the user device (Oh – Paragraph [0027]: Referring to FIG. 1, the device 100 includes a touch screen 190 and a touch screen controller 195. Also, the device 100 includes a controller 110, the mobile communication module 120, the sub-communication module 130, a multimedia module 140, a camera module 150, a GPS module 155, an input/output module 160, a sensor module 170, a storage unit 175, and a power supply unit 180; [0060]: The electronic device 100 extracts a facial region in operation 206 when the request for canceling the lock mode is input. That is, the electronic device 100 captures an image using one of the first camera 151 and the second camera 152, and extracts the facial region from the captured image; [0061]: When the facial region is extracted, the electronic device 100 proceeds with a facial recognition process 210 corresponding to steps 212 through 216, and proceeds with a gaze recognition process 220 corresponding to steps 222 through 226; [0062]: In the facial recognition process 210, the electronic device 100 extracts component information associated with the facial region in step 212. For example, the electronic device 100 extracts, from the facial region, facial component information such as a symmetric composition, a shape, hair, a color of eyes, muscles of a face, and the like; [0063]: In step 214, the electronic device 100 compares the extracted facial component information to a face registered in advance. In this scenario, the registered face may be facial component information associated with a facial region stored in advance by the user after capturing a user's face; [0064]: In step 216, the electronic device 100 determines whether the facial region is identical to the registered face. When the facial region is identical to the registered face, the electronic device 100 proceeds to step 230; [0065]: In the gaze recognition process 220, the electronic device 100 extracts an eye region from the extracted facial region in step 222. In step 224, the electronic device 100 recognizes a gaze by analyzing a pupil and a reflected light in the extracted eye region, and determines a point at which the recognized gaze is aimed. In this example, the electronic device 100 may display a gaze image 312 to represent the recognized gaze, as shown in the FIG. 3A; [0067]: After performing the facial recognition process 210 and the gaze recognition process 220 as described above, the electronic device 100 performs unlocking based on a result of the facial recognition process 210 indicating that faces are identical, and simultaneously, executes the function selected as a result of the gaze recognition process 220); bind the second set of facial features with a second particular application of the plurality of applications of the user device, wherein binding the second set of facial features with the second particular application causes the second particular application to be surfaced for display on the user device upon unlocking of the user device when the second set of facial features is detected during the unlock signal detection process of the user device (Oh – Paragraph [0058]: For example, the electronic device 100 may display a lock screen 310 and at least one icon 314-1 through 314-4 representing a function or an application in a lock mode, as shown in FIG. 3A. The at least one icon may include, for example, a music play application icon 314-1, a call function icon 314-2, a camera function icon 314-3, and an Internet browser icon 314-4. In addition to the icons, icons for other functions or applications may be included; [0066]: The electronic device 100 temporarily stores a function placed on the point at which the gaze is aimed in step 226. For example, when the recognized gaze is aimed at the Internet browser icon 314-4 as shown in the FIG. 3A, the electronic device 100 selects an Internet browser function, and temporarily stores the selected function. The electronic device 100 may select a function placed on the point at which the recognized gaze is aimed at, and temporarily stores the selected function, and proceeds to step 230; [0067]: After performing the facial recognition process 210 and the gaze recognition process 220 as described above, the electronic device 100 performs unlocking based on a result of the facial recognition process 210 indicating that faces are identical, and simultaneously, executes the function selected as a result of the gaze recognition process 220; [0068]: For example, when the facial recognition shows that the recognized face is identical to the registered face and an internet browser function (or application) is selected through the gaze recognition, the electronic device 100 performs unlocking and simultaneously executes the Internet browser function, and displays an Internet browser screen 320, as shown in the FIG. 3B).
Paul further teaches wherein the instructions are executable by the one or more processors to further configure the system to (Paul – Paragraph [0007]: In accordance with some embodiments a non-transitory computer readable storage is described. The non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of a computer system).
The motivation to combine the arts is the same as that of Claim 5.
Regarding Claim 7:
The combination of Oh, Parshionikar, Paul, Gupta, and Lin teaches the system of claim 6.
Oh further teaches wherein the second set of facial features comprises the set of unlock facial features and an additional set of intent facial features (Oh – Paragraph [0060]: The electronic device 100 extracts a facial region in operation 206 when the request for canceling the lock mode is input. That is, the electronic device 100 captures an image using one of the first camera 151 and the second camera 152, and extracts the facial region from the captured image; [0061]: When the facial region is extracted, the electronic device 100 proceeds with a facial recognition process 210 corresponding to steps 212 through 216, and proceeds with a gaze recognition process 220 corresponding to steps 222 through 226; [0062]: In the facial recognition process 210, the electronic device 100 extracts component information associated with the facial region in step 212. For example, the electronic device 100 extracts, from the facial region, facial component information such as a symmetric composition, a shape, hair, a color of eyes, muscles of a face, and the like; [0063]: In step 214, the electronic device 100 compares the extracted facial component information to a face registered in advance. In this scenario, the registered face may be facial component information associated with a facial region stored in advance by the user after capturing a user's face; [0064]: In step 216, the electronic device 100 determines whether the facial region is identical to the registered face. When the facial region is identical to the registered face, the electronic device 100 proceeds to step 230; [0065]: In the gaze recognition process 220, the electronic device 100 extracts an eye region from the extracted facial region in step 222. In step 224, the electronic device 100 recognizes a gaze by analyzing a pupil and a reflected light in the extracted eye region, and determines a point at which the recognized gaze is aimed. In this example, the electronic device 100 may display a gaze image 312 to represent the recognized gaze, as shown in the FIG. 3A; [0067]: After performing the facial recognition process 210 and the gaze recognition process 220 as described above, the electronic device 100 performs unlocking based on a result of the facial recognition process 210 indicating that faces are identical, and simultaneously, executes the function selected as a result of the gaze recognition process 220).
The motivation to combine the arts is the same as that of Claim 5.
Regarding Claim 13:
The combination of Oh, Parshionikar, Paul, and Gupta teaches the system of claim 12.
Oh further teaches wherein the set of unlock facial features is detected [from a first subset of images of the set of images], and wherein the set of intent facial features is detected [from a second subset of images of the set of images] (Oh – Paragraph [0027]: Referring to FIG. 1, the device 100 includes a touch screen 190 and a touch screen controller 195. Also, the device 100 includes a controller 110, the mobile communication module 120, the sub-communication module 130, a multimedia module 140, a camera module 150, a GPS module 155, an input/output module 160, a sensor module 170, a storage unit 175, and a power supply unit 180; [0060]: The electronic device 100 extracts a facial region in operation 206 when the request for canceling the lock mode is input. That is, the electronic device 100 captures an image using one of the first camera 151 and the second camera 152, and extracts the facial region from the captured image; [0061]: When the facial region is extracted, the electronic device 100 proceeds with a facial recognition process 210 corresponding to steps 212 through 216, and proceeds with a gaze recognition process 220 corresponding to steps 222 through 226; [0062]: In the facial recognition process 210, the electronic device 100 extracts component information associated with the facial region in step 212. For example, the electronic device 100 extracts, from the facial region, facial component information such as a symmetric composition, a shape, hair, a color of eyes, muscles of a face, and the like; [0063]: In step 214, the electronic device 100 compares the extracted facial component information to a face registered in advance. In this scenario, the registered face may be facial component information associated with a facial region stored in advance by the user after capturing a user's face; [0064]: In step 216, the electronic device 100 determines whether the facial region is identical to the registered face. When the facial region is identical to the registered face, the electronic device 100 proceeds to step 230; [0065]: In the gaze recognition process 220, the electronic device 100 extracts an eye region from the extracted facial region in step 222. In step 224, the electronic device 100 recognizes a gaze by analyzing a pupil and a reflected light in the extracted eye region, and determines a point at which the recognized gaze is aimed. In this example, the electronic device 100 may display a gaze image 312 to represent the recognized gaze, as shown in the FIG. 3A; [0067]: After performing the facial recognition process 210 and the gaze recognition process 220 as described above, the electronic device 100 performs unlocking based on a result of the facial recognition process 210 indicating that faces are identical, and simultaneously, executes the function selected as a result of the gaze recognition process 220).
The combination of Oh, Parshionikar, Paul, and Gupta does not expressly teach wherein the set of unlock facial features is detected from a first subset of images of the set of images, and wherein the set of intent facial features is detected from a second subset of images of the set of images.
However, Lin teaches wherein the set of [unlock] facial features is detected from a first subset of images of the set of images, and wherein the set of [intent] facial features is detected from a second subset of images of the set of images (Lin – Paragraph [0035]: A facial feature point is a point in an image that represents a facial feature, for example, a point that represents a facial feature or a face contour feature, and is usually represented in a form of coordinates. Samples can be obtained in various ways. For example, to ensure reliability of a sample set and improve accuracy of a preset error model, a plurality of pairs of adjacent frames of images may be extracted from a video including a face; [0034]: where each sample in the sample set includes facial feature points of a first image (or first reference image) that is a former one and facial feature points of the second image (or second reference image) that is a latter one in adjacent frames of images (or adjacent reference images)).
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to modify Oh, Parshionikar, Paul, and Gupta, further incorporating Lin to arrive at the conclusion of the claimed invention. One would be motivated to incorporate Lin’s teaching to analyze different parts of different image frames to determine verifiable facial features of a user into Oh, Parshionikar, Paul, and Gupta’s combined system for collecting and associating user facial features/gestures with device functions. This combined functionality would further enhance the system’s precision in gathering facial features/expressions to associate with user intent.
Regarding Claim 14:
The combination of Oh, Parshionikar, Paul, Gupta, and Lin teaches the system of claim 13.
Paul further teaches wherein the set of images comprises a set of image frames of a video segment captured by the one or more sensors of the user device, [and wherein the second subset of images comprises a remaining subset of image frames of the video segment that is temporally subsequent to a particular image frame of the video segment from which the set of unlock facial features is detected] (Paul – Paragraph [0101]: Device 100 optionally also includes one or more optical sensors 164. FIG. 1A shows an optical sensor coupled to optical sensor controller 158 in 1/0 subsystem 106. Optical sensor 164 optionally includes charge-coupled device (CCD) or complementary metal-oxide semiconductor (CMOS) phototransistors. Optical sensor 164 receives light from the environment, projected through one or more lenses, and converts the light to data representing an image. In conjunction with imaging module 143 (also called a camera module), optical sensor 164 optionally captures still images or video; [0103]: In some embodiments, a depth map (e.g., depth map image) contains information (e.g., values) that relates to the distance of objects in a scene from a viewpoint (e.g., a camera, an optical sensor, a depth camera sensor). In one embodiment of a depth map, each depth pixel defines the position in the viewpoint's Z-axis where its corresponding two-dimensional pixel is located. In some embodiments, a depth map is composed of pixels wherein each pixel is defined by a value (e.g., 0-255). For example, the “0” value I” represents pixels that are located at the most distant place in a “three dimensional” scene and the “255” value represents pixels that are located closest to a viewpoint (e.g., a camera, an optical sensor, a depth camera sensor) in the “three dimensional” scene. In other embodiments, a depth map represents the distance between an object in a scene and the plane of the viewpoint. In some embodiments, the depth map includes information about the relative depth of various features of an object of interest in view of the depth camera (e.g., the relative depth of eyes, nose, mouth, ears of a user's face)).
Lin further teaches wherein the set of images comprises a set of image frames of a video segment captured by the one or more sensors of the user device, and wherein the second subset of images comprises a remaining subset of image frames of the video segment that is temporally subsequent to a particular image frame of the video segment from which the set of [unlock] facial features is detected (Lin – Paragraph [0035]: pairs of adjacent frames of images may be extracted from a video including a face, and facial feature points manually marked in the plurality of pairs of adjacent frames of images are obtained as samples, where a pair of adjacent frames of images includes two adjacent images in the video; Examiner’s Comment: This teaching from Lin demonstrates a capability to gather facial features from different subsets of different sets of images captured sequentially by some optical sensor).
The motivation to combine the arts is the same as that of Claim 13.
Claim(s) 16, 17, and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Oh in view of Parshionikar.
Regarding Claim 16:
Oh teaches a system comprising: one or more processors (Oh – Paragraph [0027]: Referring to FIG. 1, the device 100 includes a touch screen 190 and a touch screen controller 195. Also, the device 100 includes a controller 110; [0028]: The controller 110 may include a CPU 111); one or more hardware storage devices storing instructions that are executable by the one or more processors to configure the system to (Oh – Paragraph [0028]: The controller 110 may include a CPU 111, a ROM 112 that stores a control program for controlling the device 100, and a RAM 113 that stores a signal or data input from the outside of the device 100 or is used as a memory region for an operation performed in the device 100. The CPU 111 may include a single-core, a dual-core, a triple-core, or a quad-core processor; and Paragraph [0099]: The memory may be an example of machine-readable storage media that are suitable for storing a program including instructions to implement the embodiments, or programs. Therefore, the invention may include a program including a code to implement an apparatus or a method as claimed herein, and a machine-readable storage medium including the program, for example, a computer-readable storage medium); detect a first set of facial features of a user using one or more sensors of the user device (Oh – Paragraph [0027]: Referring to FIG. 1, the device 100 includes a touch screen 190 and a touch screen controller 195. Also, the device 100 includes a controller 110, the mobile communication module 120, the subcommunication module 130, a multimedia module 140, a camera module 150, a GPS module 155, an input/output module 160, a sensor module 170, a storage unit 175, and a power supply unit 180; [0060]: The electronic device 100 extracts a facial region in operation 206 when the request for canceling the lock mode is input. That is, the electronic device 100 captures an image using one of the first camera 151 and the second camera 152, and extracts the facial region from the captured image; [0061]: When the facial region is extracted, the electronic device 100 proceeds with a facial recognition process 210 corresponding to steps 212 through 216, and proceeds with a gaze recognition process 220 corresponding to steps 222 through 226; [0065]: In the gaze recognition process 220, the electronic device 100 extracts an eye region from the extracted facial region in step 222. In step 224, the electronic device 100 recognizes a gaze by analyzing a pupil and a reflected light in the extracted eye region, and determines a point at which the recognized gaze is aimed. In this example, the electronic device 100 may display a gaze image 312 to represent the recognized gaze, as shown in the FIG. 3A); bind the first set of facial features with a first application of a plurality of applications of the user device, wherein binding the first set of facial features with the first application causes the first application to be surfaced for display on the user device upon unlocking of the user device when the first set of facial features … is detected during an unlock signal detection process of the user device (Oh – Paragraph [0066]: The electronic device 100 temporarily stores a function placed on the point at which the gaze is aimed in step 226. For example, when the recognized gaze is aimed at the Internet browser icon 314-4 as shown in the FIG. 3A, the electronic device 100 selects an Internet browser function, and temporarily stores the selected function. The electronic device 100 may select a function placed on the point at which the recognized gaze is aimed at, and temporarily stores the selected function, and proceeds to step 230; [0067]: After performing the facial recognition process 210 and the gaze recognition process 220 as described above, the electronic device 100 performs unlocking based on a result of the facial recognition process 210 indicating that faces are identical, and simultaneously, executes the function selected as a result of the gaze recognition process 220; [0068]: For example, when the facial recognition shows that the recognized face is identical to the registered face and an internet browser function (or application) is selected through the gaze recognition, the electronic device 100 performs unlocking and simultaneously executes the Internet browser function, and displays an Internet browser screen 320, as shown in the FIG. 3B); detect a second set of facial features of the user using the one or more sensors of the user device (Oh – Paragraph [0027]: Referring to FIG. 1, the device 100 includes a touch screen 190 and a touch screen controller 195. Also, the device 100 includes a controller 110, the mobile communication module 120, the sub-communication module 130, a multimedia module 140, a camera module 150, a GPS module 155, an input/output module 160, a sensor module 170, a storage unit 175, and a power supply unit 180; [0058]: For example, the electronic device 100 may display a lock screen 310 and at least one icon 314-1 through 314-4 representing a function or an application in a lock mode, as shown in FIG. 3A. The at least one icon may include, for example, a music play application icon 314-1, a call function icon 314-2, a camera function icon 314-3, and an Internet browser icon 314-4. In addition to the icons, icons for other functions or applications may be included; [0060]: The electronic device 100 extracts a facial region in operation 206 when the request for canceling the lock mode is input. That is, the electronic device 100 captures an image using one of the first camera 151 and the second camera 152, and extracts the facial region from the captured image; [0061]: When the facial region is extracted, the electronic device 100 proceeds with a facial recognition process 210 corresponding to steps 212 through 216, and proceeds with a gaze recognition process 220 corresponding to steps 222 through 226; [0065]: In the gaze recognition process 220, the electronic device 100 extracts an eye region from the extracted facial region in step 222. In step 224, the electronic device 100 recognizes a gaze by analyzing a pupil and a reflected light in the extracted eye region, and determines a point at which the recognized gaze is aimed. In this example, the electronic device 100 may display a gaze image 312 to represent the recognized gaze, as shown in the FIG. 3A); bind the second set of facial features with a second application of the plurality of applications of the user device, wherein binding the second set of facial features with the second application causes the second application to be surfaced for display on the user device upon unlocking of the user device when the second set of facial features … is detected during the unlock signal detection process of the user device (Oh – Paragraph [0058]: For example, the electronic device 100 may display a lock screen 310 and at least one icon 314-1 through 314-4 representing a function or an application in a lock mode, as shown in FIG. 3A. The at least one icon may include, for example, a music play application icon 314-1, a call function icon 314-2, a camera function icon 314-3, and an Internet browser icon 314-4. In addition to the icons, icons for other functions or applications may be included; [0066]: The electronic device 100 temporarily stores a function placed on the point at which the gaze is aimed in step 226. For example, when the recognized gaze is aimed at the Internet browser icon 314-4 as shown in the FIG. 3A, the electronic device 100 selects an Internet browser function, and temporarily stores the selected function. The electronic device 100 may select a function placed on the point at which the recognized gaze is aimed at, and temporarily stores the selected function, and proceeds to step 230; [0067]: After performing the facial recognition process 210 and the gaze recognition process 220 as described above, the electronic device 100 performs unlocking based on a result of the facial recognition process 210 indicating that faces are identical, and simultaneously, executes the function selected as a result of the gaze recognition process 220; [0068]: For example, when the facial recognition shows that the recognized face is identical to the registered face and an internet browser function (or application) is selected through the gaze recognition, the electronic device 100 performs unlocking and simultaneously executes the Internet browser function, and displays an Internet browser screen 320, as shown in the FIG. 3B).
Oh does not expressly teach wherein a first distance between a first facial feature of the user and a second facial feature of the user is represented in the first set of facial features; the first set of facial features capturing the first distance between the first facial feature and the second facial feature of the user; wherein a second distance between the first facial feature and the second facial feature of the user is represented in the second set of facial features, wherein the second distance is different from the first distance; the second set of facial features capturing the first distance between the first facial feature and the second facial feature of the user.
However, Parshionikar teaches wherein a first distance between a first facial feature of the user and a second facial feature of the user is represented in the first set of facial features (Parshionikar – Paragraph [0137]: For example, to detect the facial expression of a smile, the mouth can be considered to be a key feature and various points of interest on the mouth can be tracked in relation to each other as well as to the positions they were in during the calibration/initialization process. The change in position of corners of mouth relative to each other and/or center of the mouth can provide an indication of level of smile being expressed by the user. Typically, the mouth corners move away from each other when a user smiles. Such changes in position of the corners can be used to determine the level of smile or other facial expressions involving the mouth. As an example, if the distance between two corners of mouth during calibration/initialization was d1, whereas the distance between the two corner changes to d2 during a facial expression involving the mouth, then magnitude (level) of that expression can be calculated as following. Magnitude=(d2−d1)*100/d1; and Paragraph [0138]: Many other such formulae based on combination of location of points of interest on the user's face (such corners of mouth, corners of eyes, mid points of eye lids, center of pupil of the eye, center of the chin, center of upper/lower lip, tip of the nose, nostril, start/mid/end of eye brows, etc.) can be utilized. The relative locations (distance) between various points of interest and the change in those distances when going from one point in time to another can be utilized to derive a numerical value of the magnitude of a facial expression); the first set of facial features capturing the first distance between the first facial feature and the second facial feature of the user Parshionikar – Paragraph [0137]: For example, to detect the facial expression of a smile, the mouth can be considered to be a key feature and various points of interest on the mouth can be tracked in relation to each other as well as to the positions they were in during the calibration/initialization process. The change in position of corners of mouth relative to each other and/or center of the mouth can provide an indication of level of smile being expressed by the user. Typically, the mouth corners move away from each other when a user smiles. Such changes in position of the corners can be used to determine the level of smile or other facial expressions involving the mouth. As an example, if the distance between two corners of mouth during calibration/initialization was d1, whereas the distance between the two corner changes to d2 during a facial expression involving the mouth, then magnitude (level) of that expression can be calculated as following. Magnitude=(d2−d1)*100/d1; and Paragraph [0138]: Many other such formulae based on combination of location of points of interest on the user's face (such corners of mouth, corners of eyes, mid points of eye lids, center of pupil of the eye, center of the chin, center of upper/lower lip, tip of the nose, nostril, start/mid/end of eye brows, etc.) can be utilized. The relative locations (distance) between various points of interest and the change in those distances when going from one point in time to another can be utilized to derive a numerical value of the magnitude of a facial expression; wherein a second distance between the first facial feature and the second facial feature of the user is represented in the second set of facial features, wherein the second distance is different from the first distance (Parshionikar – Paragraph [0137]: For example, to detect the facial expression of a smile, the mouth can be considered to be a key feature and various points of interest on the mouth can be tracked in relation to each other as well as to the positions they were in during the calibration/initialization process. The change in position of corners of mouth relative to each other and/or center of the mouth can provide an indication of level of smile being expressed by the user. Typically, the mouth corners move away from each other when a user smiles. Such changes in position of the corners can be used to determine the level of smile or other facial expressions involving the mouth. As an example, if the distance between two corners of mouth during calibration/initialization was d1, whereas the distance between the two corner changes to d2 during a facial expression involving the mouth, then magnitude (level) of that expression can be calculated as following. Magnitude=(d2−d1)*100/d1; and Paragraph [0138]: Many other such formulae based on combination of location of points of interest on the user's face (such corners of mouth, corners of eyes, mid points of eye lids, center of pupil of the eye, center of the chin, center of upper/lower lip, tip of the nose, nostril, start/mid/end of eye brows, etc.) can be utilized. The relative locations (distance) between various points of interest and the change in those distances when going from one point in time to another can be utilized to derive a numerical value of the magnitude of a facial expression); the second set of facial features capturing the first distance between the first facial feature and the second facial feature of the user (Parshionikar – Paragraph [0137]: For example, to detect the facial expression of a smile, the mouth can be considered to be a key feature and various points of interest on the mouth can be tracked in relation to each other as well as to the positions they were in during the calibration/initialization process. The change in position of corners of mouth relative to each other and/or center of the mouth can provide an indication of level of smile being expressed by the user. Typically, the mouth corners move away from each other when a user smiles. Such changes in position of the corners can be used to determine the level of smile or other facial expressions involving the mouth. As an example, if the distance between two corners of mouth during calibration/initialization was d1, whereas the distance between the two corner changes to d2 during a facial expression involving the mouth, then magnitude (level) of that expression can be calculated as following. Magnitude=(d2−d1)*100/d1; and Paragraph [0138]: Many other such formulae based on combination of location of points of interest on the user's face (such corners of mouth, corners of eyes, mid points of eye lids, center of pupil of the eye, center of the chin, center of upper/lower lip, tip of the nose, nostril, start/mid/end of eye brows, etc.) can be utilized. The relative locations (distance) between various points of interest and the change in those distances when going from one point in time to another can be utilized to derive a numerical value of the magnitude of a facial expression).
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to modify Oh, further incorporating Parshionikar to arrive at the conclusion of the claimed invention. One would be motivated to incorporate Parshionikar’s analysis of at least two facial images to evaluate changes in distances between at least two facial points of interest into Oh’s system for collecting and associating user facial features/gestures with device functions. This combination would enhance Oh’s system by providing the capability to more precisely analyze user facial expressions and changes thereof in determining gesture-based user intent.
Regarding Claim 17:
The combination of Oh and Parshionikar teaches the system of claim 16.
Oh further teaches wherein the unlocking of the user device comprises transitioning the user device from a locked state to an unlocked state (Oh – Paragraph [0009]: aspects of the present invention provide an electronic device and an unlocking method in the electronic device which simultaneously and promptly requests execution of a desired function or application before a user cancels a lock mode, and executes the function or application requested by the user at the same time as canceling the lock mode).
The motivation to combine the arts is the same as that of Claim 16.
Regarding Claim 19:
The combination of Oh and Parshionikar teaches the system of claim 16.
Oh further teaches wherein the first set of facial features comprises a set of unlock facial features and a first set of intent facial features, and wherein the second set of facial features comprises the set of unlock facial features and a second set of intent facial features (Oh – Paragraph [0058]: For example, the electronic device 100 may display a lock screen 310 and at least one icon 314-1 through 314-4 representing a function or an application in a lock mode, as shown in FIG. 3A. The at least one icon may include, for example, a music play application icon 314-1, a call function icon 314-2, a camera function icon 314-3, and an Internet browser icon 314-4. In addition to the icons, icons for other functions or applications may be included; [0060]: The electronic device 100 extracts a facial region in operation 206 when the request for canceling the lock mode is input. That is, the electronic device 100 captures an image using one of the first camera 151 and the second camera 152, and extracts the facial region from the captured image; [0061]: When the facial region is extracted, the electronic device 100 proceeds with a facial recognition process 210 corresponding to steps 212 through 216, and proceeds with a gaze recognition process 220 corresponding to steps 222 through 226; [0065]: In the gaze recognition process 220, the electronic device 100 extracts an eye region from the extracted facial region in step 222. In step 224, the electronic device 100 recognizes a gaze by analyzing a pupil and a reflected light in the extracted eye region, and determines a point at which the recognized gaze is aimed. In this example, the electronic device 100 may display a gaze image 312 to represent the recognized gaze, as shown in the FIG. 3A; [0066]: The electronic device 100 temporarily stores a function placed on the point at which the gaze is aimed in step 226. For example, when the recognized gaze is aimed at the Internet browser icon 314-4 as shown in the FIG. 3A, the electronic device 100 selects an Internet browser function, and temporarily stores the selected function. The electronic device 100 may select a function placed on the point at which the recognized gaze is aimed at, and temporarily stores the selected function, and proceeds to step 230; [0067]: After performing the facial recognition process 210 and the gaze recognition process 220 as described above, the electronic device 100 performs unlocking based on a result of the facial recognition process 210 indicating that faces are identical, and simultaneously, executes the function selected as a result of the gaze recognition process 220; [0068]: For example, when the facial recognition shows that the recognized face is identical to the registered face and an internet browser function (or application) is selected through the gaze recognition, the electronic device 100 performs unlocking and simultaneously executes the Internet browser function, and displays an Internet browser screen 320, as shown in the FIG. 3B).
The motivation to combine the arts is the same as that of Claim 16.
Claim(s) 16, 17, and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Oh in view of Parshionikar and Paul.
Regarding Claim 20:
The combination of Oh in view of Parshionikar teaches the system of claim 19.
Oh further teaches wherein the first set of facial features is extracted [from a set of image frames detected] using the one or more sensors, and wherein the set of unlock facial features is extracted [from a first subset of images of the set of image frames], and wherein the first set of intent facial features [is extracted from a second subset of images of the set of image frames] (Oh – Paragraph [0027]: Referring to FIG. 1, the device 100 includes a touch screen 190 and a touch screen controller 195. Also, the device 100 includes a controller 110, the mobile communication module 120, the sub-communication module 130, a multimedia module 140, a camera module 150, a GPS module 155, an input/output module 160, a sensor module 170, a storage unit 175, and a power supply unit 180; [0060]: The electronic device 100 extracts a facial region in operation 206 when the request for canceling the lock mode is input. That is, the electronic device 100 captures an image using one of the first camera 151 and the second camera 152, and extracts the facial region from the captured image; [0061]: When the facial region is extracted, the electronic device 100 proceeds with a facial recognition process 210 corresponding to steps 212 through 216, and proceeds with a gaze recognition process 220 corresponding to steps 222 through 226; [0062]: In the facial recognition process 210, the electronic device 100 extracts component information associated with the facial region in step 212. For example, the electronic device 100 extracts, from the facial region, facial component information such as a symmetric composition, a shape, hair, a color of eyes, muscles of a face, and the like; [0063]: In step 214, the electronic device 100 compares the extracted facial component information to a face registered in advance. In this scenario, the registered face may be facial component information associated with a facial region stored in advance by the user after capturing a user's face; [0064]: In step 216, the electronic device 100 determines whether the facial region is identical to the registered face. When the facial region is identical to the registered face, the electronic device 100 proceeds to step 230; [0065]: In the gaze recognition process 220, the electronic device 100 extracts an eye region from the extracted facial region in step 222. In step 224, the electronic device 100 recognizes a gaze by analyzing a pupil and a reflected light in the extracted eye region, and determines a point at which the recognized gaze is aimed. In this example, the electronic device 100 may display a gaze image 312 to represent the recognized gaze, as shown in the FIG. 3A; [0067]: After performing the facial recognition process 210 and the gaze recognition process 220 as described above, the electronic device 100 performs unlocking based on a result of the facial recognition process 210 indicating that faces are identical, and simultaneously, executes the function selected as a result of the gaze recognition process 220).
The combination of Oh and Parshionikar does not expressly teach wherein the first set of facial features is extracted from a set of image frames detected using the one or more sensors, and wherein the set of unlock facial features is extracted from a first subset of images of the set of image frames, and wherein the first set of intent facial features is extracted from a second subset of images of the set of image frames.
However, Paul teaches wherein the [first set of] facial features is extracted from a set of image frames detected using the one or more sensors, [and wherein the set of unlock facial features is extracted from a first subset of images of the set of image frames, and wherein the first set of intent facial features is extracted from a second subset of images of the set of image frames] (Paul – Paragraph [0101]: Device 100 optionally also includes one or more optical sensors 164. FIG. 1A shows an optical sensor coupled to optical sensor controller 158 in I/O subsystem 106. Optical sensor 164 optionally includes charge-coupled device (CCD) or complementary metal-oxide semiconductor (CMOS) phototransistors. Optical sensor 164 receives light from the environment, projected through one or more lenses, and converts the light to data representing an image. In conjunction with imaging module 143 (also called a camera module), optical sensor 164 optionally captures still images or video; [0103]: In some embodiments, a depth map (e.g., depth map image) contains information (e.g., values) that relates to the distance of objects in a scene from a viewpoint (e.g., a camera, an optical sensor, a depth camera sensor). In one embodiment of a depth map, each depth pixel defines the position in the viewpoint's Z-axis where its corresponding two dimensional pixel is located. In some embodiments, a depth map is composed of pixels wherein each pixel is defined by a value (e.g., 0-255). For example, the “0” value represents pixels that are located at the most distant place in a “three dimensional” scene and the “255” value represents pixels that are located closest to a viewpoint (e.g., a camera, an optical sensor, a depth camera sensor) in the |” “three dimensional” scene. In other embodiments, a depth map represents the distance between an object in a scene and the plane of the viewpoint. In some embodiments, the depth map includes information about the relative depth of various features of an object of interest in view of the depth camera (e.g., the relative depth of eyes, nose, mouth, ears of a user's face)).
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to modify Oh and Parshionikar, further incorporating Paul to arrive at the conclusion of the claimed invention. One would be motivated to incorporate Paul’s teaching to collect facial image data as a set of image frames into Oh and Parshionikar’s combined system for collecting and associating user facial features/gestures with device functions. This addition would further improve the precision of facial gesture/expression detection.
The combination of Oh, Parshionikar, and Paul does not expressly teach wherein the first set of facial features is extracted from a set of image frames detected using the one or more sensors, and wherein the set of unlock facial features is extracted from a first subset of images of the set of image frames, and wherein the first set of intent facial features is extracted from a second subset of images of the set of image frames.
However, Lin teaches wherein the first set of facial features is extracted from a set of image frames detected [using the one or more sensors], and wherein the set of [unlock] facial features is extracted from a first subset of images of the set of image frames, and wherein the first set of [intent] facial features is extracted from a second subset of images of the set of image frames (Lin – Paragraph [0035]: A facial feature point is a point in an image that represents a facial feature, for example, a point that represents a facial feature or a face contour feature, and is usually represented in a form of coordinates. Samples can be obtained in various ways. For example, to ensure reliability of a sample set and improve accuracy of a preset error model, a plurality of pairs of adjacent frames of images may be extracted from a video including a face; [0034]: where each sample in the sample set includes facial feature points of a first image (or first reference image) that is a former one and facial feature points of the second image (or second reference image) that is a latter one in adjacent frames of images (or adjacent reference images); Examiner’s Comment: This teaching from Lin demonstrates a capability to gather facial features from different subsets of different sets of images captured sequentially by some optical sensor).
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to modify Oh, Parshionikar, and Paul, further incorporating Lin to arrive at the conclusion of the claimed invention. One would be motivated to incorporate Lin’s teaching to analyze different parts of different image frames to determine verifiable facial features of a user into Oh, Parshionikar, and Paul’s combined system for collecting and associating user facial features/gestures with device functions. This combined functionality would further enhance the system’s precision in gathering facial features/expressions to associate with user intent.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Lee et al. (US 20140169641 A1) teaches a method of face recognition and further recognition of an “additional component” in a process of unlocking a device and optionally activating additional functionality
Liu (US 20130300650 A1) teaches use of facial expressions as inputs to a control system, each expression having a corresponding control command
Moffat et al. (US 20180164879 A1) teaches a system for detecting user facial expressions and associating them with device functionalities to perform in response to the detections
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NICHOLAS JOSEPH DILUZIO whose telephone number is (703)756-1229. The examiner can normally be reached Mon - Fri -- 7:30 AM - 5 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Yin-Chen Shaw can be reached at 571-272-8878. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/NICHOLAS JOSEPH DILUZIO/Examiner, Art Unit 2498
/YIN CHEN SHAW/Supervisory Patent Examiner, Art Unit 2498