Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 3/20/2026 has been entered.
Response to Amendment
The amendment filed on 3/20/2026 has been entered and made of record. Claims 1, 17 and 25 are amended. Claims 2, 4, 18 and 20 are cancelled. Claims 26-27 are new. Claims 1, 3, 5-17, 19 and 21-27 are pending.
Response to Arguments
Applicant’s arguments with respect to claims 1, 17 and 25 have been considered but they are moot because the arguments do not apply to the references being used in the current rejection.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 1, 3, 5-17, 19 and 21-27 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for pre-AIA the inventor(s), at the time the application was filed, had possession of the claimed invention.
Independent claims 1, 17 and 25 recite “detecting a reflected image amongst the one or more objects based on the sensor data, wherein detecting the reflected image is based on facial feature tracking and head motion analysis of the user of the electronic device by determining at least one of: (a) the head of the user as seen in the images is rotating about a vertical axis by an amount that is more than a threshold amount of rotation of the electronic device about the vertical axis; or (b) a change in orientation of the head of the user as seen in the images about a forward-facing axis of the electronic device is less than a threshold”. Applicant doesn’t provide written description of the relationship between detecting the reflected image and head rotation about a vertical axis by more than a threshold amount of rotation in the specification. For example, a mirror image is a reflective image of the user in the image. However, it is unclear how to determine a mirror image based on the amount of head rotation more than a threshold amount of rotation of the electronic device. Similarly, it is unclear how to determine a mirror image based on rotation amount of head less than a threshold. Applicant needs to provide explanation in detail to support the claim language. Dependent claims 3,5-16, 19, 21-24 and 26-27 are rejected under the similar rationale. For the purpose of the case prosecution, the limitation “detecting a reflected image amongst the one or more objects based on the sensor data, wherein detecting the reflected image is based on facial feature tracking and head motion analysis of the user of the electronic device by determining at least one of: (a) the head of the user as seen in the images is rotating about a vertical axis by an amount that is more than a threshold amount of rotation of the electronic device about the vertical axis; or (b) a change in orientation of the head of the user as seen in the images about a forward-facing axis of the electronic device is less than a threshold” is interpreted as detecting a reflected image is based on facial feature tracking and head motion analysis of the user of the electronic device by determining rotation amount of head along one or more axes with a threshold rotation amount.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3, 5-17, 19 and 21-27 are rejected under 35 U.S.C. 103 as being unpatentable over Bramwell et al. (US 11,036,987) in view of Harrises et al. (US 2019/0371028 A1) and Rad et al. (US 2018/0268601 A1).
As to Claim 1, Bramwell teaches A method comprising: at an electronic device having a processor and one or more sensors (Bramwell, C4L45-55):
obtaining sensor data from the one or more sensors of the electronic device in a physical environment that includes one or more objects, wherein the sensor data comprises images of a head of a user (Bramwell discloses “In another example, this disclosure describes a method comprising capturing image data representative of a physical environment that includes a reflective surface, a plurality of objects, and a head-mounted display (HMD) worn by a first user” in C2L45-49, see also C5L58-C6L6 and Fig 1);
detecting a reflected image amongst the one or more objects based on the sensor data (Bramwell discloses “determining, based on the image data, a map of the physical environment including position information about the HMD, the reflective surface, and each of the plurality of physical objects; determining, based on the position of the HMD and the reflective surface” in C2L49-54, see also C5L58-C6L6); and
in accordance with detecting the reflected image: determining a context associated with a use of the electronic device in the physical environment based on the sensor data, and presenting virtual content based on the context, wherein the virtual content is positioned at a three-dimensional (3D) location based on a 3D position of the reflected image (Bramwell discloses “Artificial reality system 100A may use information obtained from a real-world or physical three-dimensional (3D) environment to render artificial reality content 122A for display by HMD 112, thereby presenting the content to user 101” in C4L56-60; “In another example, this disclosure describes a method comprising capturing, by an image capture system including a head-mounted display (HMD) worn by a user, image data representative of a physical environment, wherein the physical environment includes a reflective surface and the HMD; determining, based on the image data, a map of the physical environment, wherein the map includes position information about the reflective surface and the HMD; determining, based on the position information about the reflective surface and the HMD, that the user is visible as a reflected image in the reflective surface” in C2L59-C3L2; “Further, from the perspective of HMD 112, artificial reality system 100A renders artificial reality content based upon the estimated positions and poses for user 101 and other physical objects” in C5L54-57; “Application engine 420 determines, based on a configuration, type of application, execution context, user input, attributes of user 101, or other information, that it is appropriate to generate artificial reality content that includes a hat (i.e., virtual hat 123) located on the head of user 101” in C18L39-43, see also C6L17-27, C7L48-58, C10L14-23. Here, position and pose of HMD, time of year etc. refer to context.)
Bramwell is silent on detecting reflected image based on facial feature and head motion analysis. The combination of Harrises and Rad further teaches following limitations:
wherein detecting the reflected image is based on facial feature tracking and head motion analysis of the user of the electronic device (Harrises discloses “the wearable system may analyze the images to identify one or more cues indicative of the presence of a mirror. For example, as further described below, the cues may be based on keypoints recognition, a co-motion test, a depth discontinuity, the presence of the user's head… The cues may be used individually or in combination to determine the presence of the mirror in the user's environment. These cues may also be used to track the mirror as the user moves around in his environment” in [0564]; “The wearable system may also use other techniques to identify the face. For example, the wearable system may recognize certain features of human face such as nose, mouth, ears, etc. The wearable system may use any facial recognition algorithms to implement this technique” in [0598]; “Cues Based on Movements of the User; The wearable system may also detect the presence of a mirror by acquiring the user's movement data” in [0601-0602], see also [0604].) by determining at least one of: (a) the head of the user as seen in the images is rotating about a vertical axis by an amount that is more than a threshold amount of rotation of the electronic device about the vertical axis; or (b) a change in orientation of the head of the user as seen in the images about a forward-facing axis of the electronic device is less than a threshold (Rad further discloses “The system uses a classifier (e.g., a neural network) to determine whether an image shows an object having a rotation angle within a first predetermined range (corresponding to objects on a first side of the plane of symmetry), or within a second predetermined range (corresponding to objects on a second side of the plane of symmetry… In response to determining that the object has a rotational angle within the first predetermined range, the system determines a mirror image of the object about the plane of symmetry. The mirror image corresponds to an object in the second predetermined range…” in [0029]; “a method of estimating a 3D pose of an object having an angle of rotational symmetry β is provided. The method includes obtaining an image which includes the object, and determining a rotation angle a for the object in the obtained image, wherein α is determined with respect to a predetermined orientation for the object… In certain aspects, the first predetermined range for the rotation angle a is β/2<α≤ β” in [0030]; “In a further aspect, the trained regressor is trained using a plurality of training images, wherein each of the training images is taken from a different point of view and includes the object with a respectively different pose from a plurality of poses, wherein each of the plurality of poses has a different pose rotation angle αp, wherein αp is determined with respect to the predetermined orientation for the object, and wherein each αp of the plurality of poses is in a second predetermined range. In certain aspects, the second predetermined range for the pose rotation angle αp is 0° ≤ αp ≤ β/2. In other aspects, each αp of the plurality of training images has a respective pose in a third predetermined range, where the third predetermined range for the pose rotation angle ap is 0°-Ꟙ ≤ αp ≤ (β/2)+ Ꟙ, where Ꟙ is a predetermined angle which is less than β. In some embodiments, Ꟙ is a non-zero angle whose value is a real number less than β” in [0031].)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the invention of Bramwell with the teaching of Harrises because a user usually won't be able to see his own face unless there is a reflective surface providing a reflected image of his face (Harrises, [0594]). The motivation of combing the invention of Rad is to determine a mirror image of the object in response to a determination of a rotation angle of object within a predetermined range (Rad, Abstract).
As to Claim 3, Bramwell in view of Harrises and Rad teaches The method of claim 1. The combination of Ferens further teaches wherein detecting the reflected image is based on facial recognition of the user of the electronic device (Harrises discloses “Harrises discloses “the wearable system may analyze the images to identify one or more cues indicative of the presence of a mirror” in [0564]; “Cues using facial recognition” in [0593], see also [0594].)
As to Claim 5, Bramwell in view of Harrises and Rad teaches The method of claim 1. The combination of Ferens further teaches wherein the 3D position of the reflected image is determined based on depth data from the sensor data and the 3D location of the virtual content is based on the depth data associated with the 3D position of the reflected image (Bramwell discloses “That is, in FIG. 1A, the artificial reality application constructs artificial content by tracking and computing pose information for a frame of reference for HMD 112, and uses data received from HMD 112, external sensors 190, and/or cameras 192 to capture 3D information within the real-word, physical 3D environment 122” in C5L39-45. Harrises further discloses “In some embodiments, the wearable system may use distance recognition techniques (for example, with a stereoscopic system or a depth camera) to analyze whether an object is a mirror or a picture frame… Accordingly, by determining the depth of objects appearing within the frame, the HMD system may distinguish whether the frame is associated with a mirror” in [0583], see also [0643].)
As to Claim 6, Bramwell in view of Harrises and Rad teaches The method of claim 1, wherein detecting the reflected image is based on an object detection technique using machine learning (Harrises discloses “The object recognitions may additionally or alternatively be performed by a variety of machine learning algorithms described herein” in [0423]; “If the object recognizer recognizes a set of points to be a mirror, the system may attach semantic information that the mirror has a reflective surface that may reflect images of objects in the room” in [0424]; “The wearable system may use machine learning techniques (e.g., a naive Bayes model) together with the cues described herein to determine likelihood that the object is a mirror (or an aperture).” in [0614].)
As to Claim 7, Bramwell in view of Harrises and Rad teaches The method of claim 1, wherein the context comprises a time of day, and presenting the virtual content is based on the time of day (Bramwell discloses “In some examples, map data includes different master 3D maps for the same area of a physical 3D environment, the different master 3D maps representing different states of the physical environment. For example, a first master 3D map may describe an area at a first time e.g., August 2015, while a second master 3D map may describe the area at a second time, e.g., October 2016. Application engine 320 may determine to use the first master 3D map responsive to a request from the user or responsive to a trigger within an artificial reality application, for instance.” in C16L1-10.)
As to Claim 8, Bramwell in view of Harrises and Rad teaches The method of claim 7, wherein the time of day is the morning and the virtual content comprises a calendar application or a news application (Bramwell discloses “for instance. The mapping engine 328 may indicate in map data 330 that the first master 3D map is the master 3D map that is to be used for rendering artificial reality content for an artificial reality application. In this way, an artificial reality system including console 106 can render artificial reality content using historical map data describing a physical 3D environment as it appeared in earlier times. This technique may be advantageous for education-related artificial reality applications, for instance.” in C16L10-18.)
As to Claim 9, Bramwell in view of Harrises and Rad teaches The method of claim 1, wherein the context comprises movements of the user of the electronic device with respect to the reflected image, and presenting the virtual content is based on the movements of the user (Bramwell discloses “HMD 112 may generate artificial reality content 122A. For instance, referring again to FIG. 1A and FIG. 4, mapping engine 428 outputs, to application engine 420, information about mapping information for physical environment 120A. Pose tracker 426 outputs, to application engine 420, information about the current pose determined for a frame of reference of HMD 112. Application engine 420 determines, based on the mapping and pose information, that from the perspective of user 101, HMD 112 is visible in mirror 109 (i.e., user 101 can see HMD 112 reflected by mirror 109). Application engine 420 determines, based on a configuration, type of application, execution context, user input, attributes of user 101, or other information, that it is appropriate to generate artificial reality content that includes a hat (i.e., virtual hat 123) located on the head of user 101” in C18L29-43.)
As to Claim 10, Bramwell in view of Harrises and Rad teaches The method of claim 1, wherein the context comprises a user interaction with the reflected image, and presenting the virtual content is based on the user interaction with the reflected image (Bramwell discloses “Responsive to determining that user 101 is positioned in front of mirror 109, console 106 generates artificial reality content 122B, and includes user interface menu 124 within artificial reality content 122B. In the example of FIG. 1B, user interface menu 124 further includes user interface elements 126. Console 106 causes artificial reality content 122B to be presented to user 101 within HMD 112. In the example described, artificial reality content 122B may be presented to user 101 within HMD 112 simply in response to detecting that user 101 is positioned in front of mirror 109. C7L48-58, see also C10L14-23, C18L29-47.)
As to Claim 11, Bramwell in view of Harrises and Rad teaches The method of claim 1, wherein determining the context comprises determining use of the electronic device in a new location (Bramwell discloses “determining, based on the image data, a map of the physical environment, wherein the map includes position information about the reflective surface and the HMD; determining, based on the position information about the reflective surface and the HMD, that the user is visible as a reflected image in the reflective surface” in C2L64-C3L2.)
As to Claim 12, Bramwell in view of Harrises and Rad teaches The method of claim 1, wherein determining the context comprises determining use of the electronic device during a type of activity (Bramwell discloses “User interface engine 329 may perform functions relating to generating a user interface when a user is in close proximity to mirror 109 and/or when a user performs a gesture or action (e.g., touching the surface of mirror 109). User interface engine 329 may receive information from application engine 320, pose tracker 326, and/or mapping engine 328 and based on that information, generate a user interface (e.g., user interface menu 124 having user interface elements 126)” in C16L19-27.)
As to Claim 13, Bramwell in view of Harrises and Rad teaches The method of claim 1, wherein determining the context comprises determining that the electronic device is within a proximity threshold distance of a location, an object, another electronic device, or a person (Bramwell discloses “User interface engine 329 may perform functions relating to generating a user interface when a user is in close proximity to mirror 109 and/or when a user performs a gesture or action (e.g., touching the surface of mirror 109). User interface engine 329 may receive information from application engine 320, pose tracker 326, and/or mapping engine 328 and based on that information, generate a user interface (e.g., user interface menu 124 having user interface elements 126)” in C16L19-27.)
As to Claim 14, Bramwell in view of Harrises and Rad teaches The method of claim 1, wherein a depth position of the 3D location of the virtual content is the same as a depth of a reflected object detected in the reflected image (Bramwell discloses “Console 106 generates artificial reality content 122A, and includes virtual hat 123 at a position that is determined based on the location, within physical environment 120A, of HMD 112. In some examples, virtual hat 123 is "locked" (or substantially locked) to the head of user 101 or to HMD 112 so that when the head of user 101 or when HMD 112 moves, the appropriate position of virtual hat 123 on the reflected image of the head of user 101 would also move C6L19-27, see also claim 5.)
As to Claim 15, Bramwell in view of Harrises and Rad teaches The method of claim 1, wherein the physical environment includes one or more objects, and detecting the reflected image comprises detecting a mirror amongst the one or more objects based on the sensor data (Bramwell discloses “For instance, with reference to FIG. 1A, each of HMD 112, external sensors 190, and/or cameras 192 capture images within physical environment 120A. Console 106 receives such images and determines the position of physical objects within physical environment 120A, including user 101, HMD 112, and mirror 109. Console 106 determines the region of space reflected in mirror 109…” in C5L60-67.)
As to Claim 16, Bramwell in view of Harrises and Rad teaches The method of claim 1, wherein the electronic device is a head-mounted device (HMD) (Bramwell, Fig 1-2.)
Claim 17 recites similar limitations as claim 1 but in a device form. Therefore, the same rationale used for claim 1 is applied.
Claim 19 is rejected based upon similar rationale as Claim 3.
Claim 21 is rejected based upon similar rationale as Claim 5.
Claim 22 is rejected based upon similar rationale as Claim 6.
Claim 23 is rejected based upon similar rationale as Claim 7.
Claim 24 is rejected based upon similar rationale as Claim 8.
Claim 25 recites similar limitations as claim 1 but in a computer readable medium form. Therefore, the same rationale used for claim 1 is applied.
As to Claim 26, Bramwell in view of Harrises and Rad teaches The method of claim 1, wherein the forward-facing axis extends along a viewing direction of the electronic device (Rad discloses “In one example, a method for estimating a three-dimensional ("3D") pose of an object having an angle of rotational symmetry includes…” in [0007]. Here, the object in the image can be rotated along pitch, roll and Yaw direction, see also [0030-0031].)
As to Claim 27, Bramwell in view of Harrises and Rad teaches The method of claim 1, wherein the head of the user as seen in the images is rotating about the vertical axis in a yaw direction (Rad discloses “In one example, a method for estimating a three-dimensional ("3D") pose of an object having an angle of rotational symmetry includes…” in [0007]. Here, the object in the image can be rotated along pitch, roll and Yaw direction, see also [0030-0031].)
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WEIMING HE whose telephone number is (571)270-1221. The examiner can normally be reached on Monday-Friday, 8:30am-5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Tammy Goddard can be reached on 571-272-7773. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see https://ppair-my.uspto.gov/pair/PrivatePair. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WEIMING HE/
Primary Examiner, Art Unit 2611
Prosecution Insights