DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
This application is a continuation of US Application No. 18/375,058 filed on September 29, 2023, which claims the benefit of US Provisional Application No. 63/456,133 filed on March 31, 2023.
Information Disclosure Statement
The information disclosure statement (IDS) submitted is considered by the examiner.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 1, 3-22 and 25 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-13,15-23 of U.S. Patent No. 12,159,351 B2. Although the claims at issue are not identical, they are not patentably distinct from each other because claims 1, 13 and 23 are a broader recitation of independent claims 1, 14 and 25 of Patent No. ‘351. Thus, claims 1, 13 and 23 are anticipated by 1, 14 and 25 of Patent No. ‘351 as shown below.
Instant Application 18/931,135
1. A method comprising: at a wearable electronic device having a processor: obtaining first data corresponding to a current appearance of a face portion of a user, the first data based on first sensor data captured via one or more sensors; determining that the user is different than another user associated with a user appearance enrollment for the device; in accordance with determining that the user is different than the another user associated with a user appearance enrollment for the device, determining to use second data corresponding to a characteristic of the user; obtaining the second data corresponding to the characteristic of the user; generating a representation of the face portion based on the first data and the second data; and presenting the representation of the face portion on an outward-facing display of the wearable electronic device.
14. A wearable electronic device comprising: a non-transitory computer-readable storage medium; one or more sensors; and one or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium comprises program instructions that, when executed on the one or more processors, cause the one or more processors to perform operations comprising: obtaining first data corresponding to a current appearance of a face portion of a user, the first data based on first sensor data captured via one or more sensors; determining that the user is different than another user associated with a user appearance enrollment for the device; in accordance with determining that the user is different than the another user associated with a user appearance enrollment for the device, determining to use second data corresponding to a characteristic of the user; obtaining the second data corresponding to the characteristic of the user; generating a representation of the face portion based on the first data and the second data; and presenting the representation of the face portion on an outward-facing display of the wearable electronic device.
25. A non-transitory computer-readable storage medium, storing program instructions executable on a device including one or more processors to perform operations comprising: obtaining first data corresponding to a current appearance of a face portion of a user, the first data based on first sensor data captured via one or more sensors; determining that the user is different than another user associated with a user appearance enrollment for the device; in accordance with determining that the user is different than the another user associated with a user appearance enrollment for the device, determining to use second data corresponding to a characteristic of the user; obtaining the second data corresponding to the characteristic of the user; generating a representation of the face portion based on the first data and the second data; and presenting the representation of the face portion on an outward-facing display of the wearable electronic device.
Patent No. 12,159,351 B2
1. A method comprising: at a wearable electronic device having a processor: obtaining first data corresponding to a current appearance of a face portion of a user, the first data based on first sensor data captured via one or more sensors; determining that the user is different than another user associated with a user appearance enrollment for the device; in accordance with determining that the user is different than the another user associated with a user appearance enrollment for the device, determining to use second data corresponding to a skin tone value and an interpupillary distance (IPD) value of the user; obtaining the second data corresponding to the skin tone value and the IPD value of the user; generating a representation of the face portion based on the first data and the second data; and presenting the representation of the face portion on an outward-facing display of the wearable electronic device.
13. A wearable electronic device comprising: a non-transitory computer-readable storage medium; one or more sensors; and one or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium comprises program instructions that, when executed on the one or more processors, cause the one or more processors to perform operations comprising: obtaining first data corresponding to a current appearance of a face portion of a user, the first data based on first sensor data captured via one or more sensors; determining that the user is different than another user associated with a user appearance enrollment for the device; in accordance with determining that the user is different than the another user associated with a user appearance enrollment for the device, determining to use second data corresponding to a skin tone value and an interpupillary distance (IPD) value of the user; obtaining the second data corresponding to the skin tone value and the IPD value of the user; generating a representation of the face portion based on the first data and the second data; and presenting the representation of the face portion on an outward-facing display of the wearable electronic device.
23. A non-transitory computer-readable storage medium, storing program instructions executable on a device including one or more processors to perform operations comprising: obtaining first data corresponding to a current appearance of a face portion of a user, the first data based on first sensor data captured via one or more sensors; determining that the user is different than another user associated with a user appearance enrollment for the device; in accordance with determining that the user is different than the another user associated with a user appearance enrollment for the device, determining to use second data corresponding to a skin tone value and an interpupillary distance (IPD) value of the user; obtaining the second data corresponding to the skin tone value and the IPD value of the user; generating a representation of the face portion based on the first data and the second data; and presenting the representation of the face portion on an outward-facing display of a wearable electronic device.
Dependent claims 3-13 and 15-22 are also anticipated by claims 2-12 and 15-22 of Patent No. ‘351. Thus, claims 3-13 and 15-22 are rejected for the same reason as the independent claims 1 and 14.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-5, 12-16, 23-25 is/are rejected under 35 U.S.C. 103 as being unpatentable over Chen (US Publication Number 2018/0004478 A1) in view of Larson et al. (US Patent Number 10,693,872 B1, hereinafter “Larson”).
(1) regarding claim 1:
As shown in fig. 6A-6B, Chen disclosed a method (para. [0020], note that a method that is used for presenting the expressions of a HMD user while the HMD user is interacting with content) comprising:
at a wearable electronic device having a processor (para. [0026], note that user 100 is shown wearing a head-mounted display (HMD) 102. Note that the head-mounted display 102 includes a processor 1300, para. [0073]):
obtaining first data corresponding to a current appearance of a face portion of a user, the first data based on first sensor data captured via one or more sensors (para. [0049], note that the video rendering module 322 receives the image or video of a portion of the user's facial features that are behind the HMD from one or more image or video capturing devices);
obtaining the second data corresponding to the characteristic of the user (para. [0082], note that bio-sensors 1342 are included to enable detection of physiological data from a user. In one embodiment, the bio-sensors 1342 include one or more dry electrodes for detecting bio-electric signals of the user through the user's skin);
generating a representation of the face portion based on the first data and the second data (para. [0050], note that the image rendering module 323 then uses this analysis to carefully map the portion of the facial features captured by the image capturing device(s) and to generate image overlay data. Similar analysis may be done by the video rendering module 322 to generate the image overlay data); and
presenting the representation of the face portion on an outward-facing display of the wearable electronic device (para. [0066], note that the image overlay data is forwarded to the second display screen of the HMD for rendering, as illustrated in operation 640. The image overlay data is captured and rendered while the user is watching content on a first display screen of the HMD).
Chen disclosed most of the subject matter as described as above except for specifically teaching determining that the user is different than another user associated with a user appearance enrollment for the device; in accordance with determining that the user is different than the another user associated with a user appearance enrollment for the device, determining to use second data corresponding to a characteristic of the user.
However, Larson disclosed determining that the user is different than another user associated with a user appearance enrollment for the device (col. 3, lines 64-col. 4, line 2, note that the collected biographic data is run against multiple attributes and/or variables to verify that the biographic information collected during the enrollment is accurate and/or to determine a probability that the enrollee identity is a synthetic identity); in accordance with determining that the user is different than the another user associated with a user appearance enrollment for the device, determining to use second data corresponding to a characteristic of the user (col. 27, lines 30-40, note that although the example of FIG. 2A only uses two different biometrics to authenticate the enrollee's identity, in other embodiments, any number and combination of biometric data may be collected and used to authenticate the enrollee's identity. For example, the secondary biometric data collected at operation 205 (or tertiary biometric data collected before or after the secondary biometric data (not shown by FIG. 2A)) may be palm/hand image data, which may be compared with stored palm/hand images in a same or similar manner as the facial image data discussed previously).
At the time of filing for the invention, it would have been obvious to a person of ordinary skilled in the art to teach determining that the user is different than another user associated with a user appearance enrollment for the device; in accordance with determining that the user is different than the another user associated with a user appearance enrollment for the device, determining to use second data corresponding to a characteristic of the user. The suggestion/motivation for doing so would have been in order to provide identity management and identity verification in which users authenticate their identities during an enrollment process, and may access and modify their identity information via a secure portal (abs.). Therefore, it would have been obvious to combine Chen with Larson to obtain the invention as specified in claim 1.
(2) regarding claim 2:
Chen further disclosed the method of claim 1, wherein the characteristic of the user corresponds to skin of the user (para. [0082], note that he bio-sensors 1342 include one or more dry electrodes for detecting bio-electric signals of the user through the user's skin).
(3) regarding claim 3:
Chen disclosed most of the subject matter as described as above except for specifically teaching wherein the characteristic of the user is based on one or more images of hands of the user.
However, Larson disclosed wherein the characteristic of the user is based on one or more images of hands of the user (col. 27, lines 30-40, note that although the example of FIG. 2A only uses two different biometrics to authenticate the enrollee's identity, in other embodiments, any number and combination of biometric data may be collected and used to authenticate the enrollee's identity. For example, the secondary biometric data collected at operation 205 (or tertiary biometric data collected before or after the secondary biometric data (not shown by FIG. 2A)) may be palm/hand image data).
At the time of filing for the invention, it would have been obvious to a person of ordinary skilled in the art to teach wherein the characteristic of the user is based on one or more images of hands of the user. The suggestion/motivation for doing so would have been in order to provide identity management and identity verification in which users authenticate their identities during an enrollment process, and may access and modify their identity information via a secure portal (abs.). Therefore, it would have been obvious to combine Chen with Larson to obtain the invention as specified in claim 3.
(4) regarding claim 4:
Chen disclosed most of the subject matter as described as above except for specifically teaching wherein the one or more images of the hands are captured during a hand enrollment.
However, Larson disclosed wherein the one or more images of the hands are captured during a hand enrollment (col. 37, lines 1-4, note that the GUI instance 905 also includes a GCE 925, which when selected by the enrollee, proceeds to GUI instance 910 that is used to perform a palm scan of the enrollee's other hand by aligning the other palm/hand within the outline 935 in a same or similar manner as discussed previously. Additionally, the GUI instance 910 includes a text area 930 to indicate the particular hand/palm that should be captured (e.g., left or right palm/hand).).
At the time of filing for the invention, it would have been obvious to a person of ordinary skilled in the art to teach wherein the one or more images of the hands are captured during a hand enrollment. The suggestion/motivation for doing so would have been in order to provide identity management and identity verification in which users authenticate their identities during an enrollment process, and may access and modify their identity information via a secure portal (abs.). Therefore, it would have been obvious to combine Chen with Larson to obtain the invention as specified in claim 4.
(5) regarding claim 5:
Chen further disclosed the method of claim 1, wherein the characteristic of the user corresponds to a portion of the face portion not depicted in the first sensor data (para. [0082], note that Bio-sensors 1342 are included to enable detection of physiological data from a user. In one embodiment, the bio-sensors 1342 include one or more dry electrodes for detecting bio-electric signals of the user through the user's skin).
(6) regarding claim 12:
Chen further disclosed the method of claim 1 further comprising providing different views of the face portion over time based on a changing position of an observer, wherein the wearable electronic device provides the views via a lenticular display (para. [0054], note that the superimposition allows the non-HMD user to observe the HMD user's current expressions, as though the HMD user is not wearing the HMD. The capturing, processing and provisioning of the image for the portion of the user's facial features that are behind the HMD, are done in substantial real-time while the HMD user continues to view or interact with the content that is currently rendering on the first display 310. The non-HMD users may observe the changing expressions as it is being provided by the HMD user, without the HMD user having to remove his HMD).
(7) regarding claim 13:
Chen further disclosed the method of claim 1, wherein the wearable electronic device comprises a head mounted device (HMD) that blocks a direct view the face portion when the wearable electronic device is being worn (para. [0045], note that when a first user adorns the HMD, a portion of the first user's face is hidden behind the HMD).
The proposed rejection of claims 1, 3-5, 12-13 renders obvious the steps of the system claims (fig. 3) 14-16, 23-24 and the non-transitory computer-readable storage medium claim (fig. 8) 25 because these steps occur in the operation of the proposed rejection as discussed above. Thus, the arguments similar to that presented above for claims 1, 3-5, 12-13 are equally applicable to claims 14-16, 23-25.
Claim(s) 6-11 and 17-22 is/are rejected under 35 U.S.C. 103 as being unpatentable over Chen and Larson, further in view of Rekimoto et al. (NPL, “Behind-the-Mask: A Face-Through Head-Mounted Display”, 2018).
(1) regarding claim 6:
Chen disclosed most of the subject matter as described as above except for specifically teaching wherein generating the representation of the face portion comprises inputting the first data and the second data to a neural network trained to output a 3D mesh representing a shape of the face portion and an image representing a texture of the face portion.
However, Rekimoto disclosed wherein generating the representation of the face portion comprises inputting the first data and the second data to a neural network trained to output a 3D mesh representing a shape of the face portion and an image representing a texture of the face portion (fig. 7, note that the image is colorized by the neural network. Also see page 4, 3.4 Image Colorization, para. [0003], note that this method uses a deep convolutional neural network for colorization. A large data set of color images and corresponding grayscale images is used for training. Once the training is completed, the input grayscale image can be colorized. Also see para. [0006], note that Figure 8 shows the final results of the face reconstruction. The 3D face model with moving eyes and mouth can be reconstructed).
At the time of filing for the invention, it would have been obvious to a person of ordinary skilled in the art to teach wherein generating the representation of the face portion comprises inputting the first data and the second data to a neural network trained to output a 3D mesh representing a shape of the face portion and an image representing a texture of the face portion. The suggestion/motivation for doing so would have been in order to acquire the user’s facial expressions and gaze: features that are also gaining attention in HMD technologies (page 2, para. [0001]). Therefore, it would have been obvious to combine Chen, Larson and Rekimoto to obtain the invention as specified in claim 6.
(2) regarding claim 7:
Chen disclosed most of the subject matter as described as above except for specifically teaching inputting live eye camera data and neutral eye camera data to the neural network.
However, Rekimoto disclosed inputting live eye camera data and neutral eye camera data to the neural network (figs. 5, page 3, para. [0003], note that Figure 5 shows the examples of captured images. This IR-cut filter and side-camera configuration is inspired by the eye-tracking system in [5]. By using our proposed system, it should also be possible to measure the gaze direction and eye information (e.g., blinking). Moreover, this information would be useful for enhancing VR applications).
At the time of filing for the invention, it would have been obvious to a person of ordinary skilled in the art to teach inputting live eye camera data and neutral eye camera data to the neural network. The suggestion/motivation for doing so would have been in order to acquire the user’s facial expressions and gaze: features that are also gaining attention in HMD technologies (page 2, para. [0001]). Therefore, it would have been obvious to combine Chen, Larson and Rekimoto to obtain the invention as specified in claim 7.
(3) regarding claim 8:
Chen disclosed most of the subject matter as described as above except for specifically teaching wherein generating the representation of the face portion comprises determining a change based on the live eye camera data and the neutral eye camera data.
However, Rekimoto disclosed wherein generating the representation of the face portion comprises determining a change based on the live eye camera data and the neutral eye camera data (figs. 5, page 3, para. [0004], note that To fit an image onto the face model (and the corresponding face texture image), we used the following calibration method. We first show the 3D face model and camera pictures (from one of the eyes and the mouth cameras) on the same window (Figure 6). We then rotate, translate, and change the field-of-view of the 3D projection for the 3D face model to match the 3D model and corresponding eye (or mouth) image. This process is equivalent to determine the position and orientation of the eye or mouth camera against the user’s head).
At the time of filing for the invention, it would have been obvious to a person of ordinary skilled in the art to teach wherein generating the representation of the face portion comprises determining a change based on the live eye camera data and the neutral eye camera data. The suggestion/motivation for doing so would have been in order to acquire the user’s facial expressions and gaze: features that are also gaining attention in HMD technologies (page 2, para. [0001]). Therefore, it would have been obvious to combine Chen, Larson and Rekimoto to obtain the invention as specified in claim 8.
(4) regarding claim 9:
Chen disclosed most of the subject matter as described as above except for specifically teaching wherein presenting the representation of the face portion on the outward-facing display of the wearable electronic device comprises: generating a live 3D face portion representation by mapping the image to the 3D mesh; and generating a view from a viewpoint of the live 3D face portion representation from a viewpoint.
However, Rekimoto disclosed wherein presenting the representation of the face portion on the outward-facing display of the wearable electronic device comprises: generating a live 3D face portion representation by mapping the image to the 3D mesh (page. 4, Result and Application, para. [0001], note that the 3D face model with moving eyes and mouth can be reconstructed. In addition, we can combine the head motion data (e.g., translation and rotation) obtained from the HMD position tracker by using this 3D face model); and generating a view from a viewpoint of the live 3D face portion representation from a viewpoint (para. [0002], note that mutual shared face-to-face communication through a shared virtual space, where participants joints virtual space and have communications with other users. Unlike avatar based shared virtual communication, participants can see other participant’s own faces and facial expressions).
At the time of filing for the invention, it would have been obvious to a person of ordinary skilled in the art to teach wherein presenting the representation of the face portion on the outward-facing display of the wearable electronic device comprises: generating a live 3D face portion representation by mapping the image to the 3D mesh. The suggestion/motivation for doing so would have been in order to acquire the user’s facial expressions and gaze: features that are also gaining attention in HMD technologies (page 2, para. [0001]). Therefore, it would have been obvious to combine Chen, Larson and Rekimoto to obtain the invention as specified in claim 9.
(5) regarding claim 10:
Chen disclosed most of the subject matter as described as above except for specifically teaching wherein the view depicts the view such that a viewer at the viewpoint observes the eyes and surrounding portions of the face portion in 3D positions corresponding to their actual 3D positions.
However, Rekimoto disclosed wherein the view depicts the view such that a viewer at the viewpoint observes the eyes and surrounding portions of the face portion in 3D positions corresponding to their actual 3D positions (page. 4, Result and Application, para. [0001], note that the 3D face model with moving eyes and mouth can be reconstructed. In addition, we can combine the head motion data (e.g., translation and rotation) obtained from the HMD position tracker by using this 3D face model).
At the time of filing for the invention, it would have been obvious to a person of ordinary skilled in the art to teach wherein the view depicts the view such that a viewer at the viewpoint observes the eyes and surrounding portions of the face portion in 3D positions corresponding to their actual 3D positions. The suggestion/motivation for doing so would have been in order to acquire the user’s facial expressions and gaze: features that are also gaining attention in HMD technologies (page 2, para. [0001]). Therefore, it would have been obvious to combine Chen, Larson and Rekimoto to obtain the invention as specified in claim 10.
(6) regarding claim 11:
Chen further disclosed the method of claim 9, wherein the viewpoint is determined based on determining a relative position of an observer to the wearable electronic device (para. [0059], note that each of the different emotions are detected by the inward facing camera or gaze detection camera and is provided for rendering the second display screen 311, so that the non-HMD user watching the HMD user can see the change in expression defining the different emotions that the HMD user goes through while interacting or viewing content on the first display screen of the HMD, see fig.5).
The proposed rejection of claims 6-11 renders obvious the steps of the system claims (fig. 3) 17-22 because these steps occur in the operation of the proposed rejection as discussed above. Thus, the arguments similar to that presented above for claims 6-11 are equally applicable to claims 17-22.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Kontsevich et al. (US Publication Number 2018/0060683 A1) disclosed an enrollment system which match a palm digital representation to a palm template. The platform includes logic causing presenting a viewfinder image including an image stream and/or a cue appearing to reside substantially in front of a user to guide the user to align a palm at a position within a field of view of a camera; capturing a set of images using the camera; processing the set of images to determine a set of identifying features of the palm; and comparing the set of identified features with enrolled palm prints to identify the palm.
Ilie et al. (US Publication Number 2019/0285881 A1) disclosed systems and methods for synthesizing an image of the face by a head-mounted display (HMD), such as an augmented reality device, are disclosed. The HMD may be able to observe only a portion of the face with an inward-facing imaging system, e.g., the periocular region. The systems and methods described herein can generate a mapping of a conformation of the portion of the face that is not imaged based at least partly on a conformation of the portion of the face that is imaged. The HMD can receive an image of a portion of the face and use the mapping to determine a conformation of the portion of the face that is not observed. The HMD can combine the observed and unobserved portions to synthesize a full-face image.
Any inquiry concerning this communication or earlier communication from the examiner should be directed to Hilina K Demeter whose telephone number is (571) 270-1676.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, King Y. Poon could be reached at (571) 270- 0728. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about PAIR system, see http://pari-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/HILINA K DEMETER/Primary Examiner, Art Unit 2617
Prosecution Insights