DETAILED ACTION
Notice of Pre-AIA or AIA Status
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Preliminary Amendment
2. The Preliminary Amendment filed on 7/22/2024 is accepted and examined below.
Claim Status
3. Claims 1-15 are canceled.
4. Claims 16-35 are newly added and pending in the present application.
Double Patenting
5. The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
6. Claims 16-17, 19-25, and 27-34 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-13 of U.S. Patent No. 12,067,686 B2 (patent 686). Although the claims at issue are not identical, they are not patentably distinct from each other because while the claims are not the same, they are similar.
7 Claims 18, 26, and 35 are rejected on the ground of nonstatutory double patenting as being unpatentable over claim 1 and 7 of U.S. Patent No. 12,067,686 B2 (patent 686) in view of Natsume et al. (US Patent Application Publication No. 2006/0253505 A1). Natsume at paragraph [0076] discloses the storing of image compilation data into an external storage device such as a server, which can correspond to a data source. Although the claims at issue are not identical, they are not patentably distinct from each other because while the claims are not the same, they are similar.
8. The following table shows correspondence between the claims of present application and claims of patent 686.
Claims of present application
16
17
18
19
20
21
22
23
24
25
Claims of patent 686
1
1
1 in view of Natsume
2
3
4
5
6
7
7
26
27
28
29
30
31
32
33
34
35
7 in view of Natsume
8
10
11
12
13
7
8
9
7 in view of Natsume
9. The following table shows correspondence between claim16 of present application and claim 1 of patent 686.
Claim 16 of present application
Claim 1 of patent 686
16. A computer-implemented method for creating Virtual Reality (VR) interactions between two or more users wherein at least one user is wearing a VR headset, comprising:
1. A computer method for creating Virtual Reality (VR) interactions between two or more users wherein at least one user is wearing a VR headset, comprising:
generating a two-dimensional (2D) image of the user wearing the VR headset using one or more camera devices located external of the VR headset;
tracking the pupil movement of the user while wearing a VR headset using one or more sensors positioned internal of the VR headset;
removing an image of the VR headset from the generated 2D image of the user wearing the headset; and
creating a two-dimensional (2D) image of the user wearing a VR headset using the one or more external camera devices;
tracking the pupil movement of the user using one or more sensors positioned internal of the VR headset;
remove a headset image from the created 2D image of the user wearing the headset;
stitching a compilation image of the user consisting of one or more captured static images of the user without wearing the VR headset, responsive to the user's tracked pupil movement, in the generated 2D image of the user, whereby another user can view a VR image of the user while wearing the VR headset while displaying near real-time pupil movement of the user while wearing the VR headset.
stitch the compilation in a 2D image of the user contingent upon the user's tracked pupil movement such that another user can view a VR image of the user
generating a compilation consisting of one or more captured static images of the at least one user without the VR headset by one or more camera devices located external of the VR headset and captured real-time pupil images of the user captured by one or more camera devices located internal of the VR headset;
whereby the compilation image is superimposed proximate to the location the VR headset image was removed from the user
whereby the compilation is superimposed proximate to the location the headset was removed from the user wearing the VR headset
as if the user was not wearing the VR headset
as if the user was not wearing the headset.
10. Claims 16-17, 19, 21, 23-25, 27, 29, and 31-33 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-8 of U.S. Patent No. 11,551,427 B2 (patent 427). Although the claims at issue are not identical, they are not patentably distinct from each other because while the claims are not the same, the claims are similar.
11. Claims 18, 26, and 35 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1 and 5 of patent 427 in view of Natsume. Natsume discloses in paragraph [0076] the storing of image compilation data into an external storage device such as a server, which can correspond to a data source. Claims 20 and 28 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1 and 5 of patent 427 in view of Malzbender (US Patent Application Publication No. 2012/0026277 A1) which discloses depth sensors. Claims 22 and 30 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1 and 5 of patent 427 in view of Arsenich (US Patent Application Publication No. 2009/0102915 A1) which discloses eye tracing micro video array sensor. Claim 34 is rejected on the ground of nonstatutory double patenting as being unpatentable over claim 1 of patent 427 in view of Deng (US Patent Application Publication No. 2016/0134803 A1) which discloses camera integrated into smartphone for capturing head/face. Although the claims at issue are not identical, they are not patentably distinct from each other because while the claims are not the same, the claims of present application are broader.
12. The following table shows correspondence between the claims of present application and claims of patent 427.
Claims of present application
16
17
18
19
20
21
22
23
24
25
Claims of patent 427
5
5
5 in view of Natsume
6
5 in view of Malzbender
7
5 in view of Arsenich
8
1
1
26
27
28
29
30
31
32
33
34
35
1 in view of Natsume
2
1 in view of Malzbender
3
1 in view of Arsenich
4
1
2
1 in view of Deng
1 in view of Natsume
13. The following table shows correspondence between claim 16 of present application and claim 5 of patent 427.
Claim 16 of present application
Claim 5 of patent 427
16. (Newly Added) A computer-implemented method for creating Virtual Reality (VR) interactions between two or more users wherein at least one user is wearing a VR headset, comprising:
generating a two-dimensional (2D) image of the user wearing the VR headset using one or more camera devices located external of the VR headset;
tracking the pupil movement of the user while wearing a VR headset using one or more sensors positioned internal of the VR headset;
5. A computer method for removing a headset image in Virtual Reality system, comprising:
positioning one or more pupil LED sensors internal of the VR headset for capturing and tracking pupil movement of the at least one user wearing the VR headset;
positioning at least one camera device external of the VR headset for capturing:
1) video image of at least one user wearing the VR headset; and
2) one or more static images of the user not wearing the VR headset;
positioning one or more camera devices internal of the VR headset for capturing and recording real-time images of a user's pupils simulatanous with the capturing and tracking of the user's pupil movement by the one or more pupil LED sensors; and
removing an image of the VR headset from the generated 2D image of the user wearing the headset;
generating a two-dimensional (2D) image of the least one user, by utilizing the recorded real-time images of the user's pupils and the simulatanoulsy captured tracked pupil movements of the user's pupils while wearing the VR headset, such that an image of the VR headset is virtually removed from the image of the at least one user wearing the VR headset by
and stitching a compilation image of the user consisting of one or more captured static images of the user without wearing the VR headset, responsive to the user's tracked pupil movement, in the generated 2D image of the user, whereby the compilation image is superimposed proximate to the location the VR headset image was removed from the user whereby another user can view a VR image of the user while wearing the VR headset as if the user was not wearing the VR headset while displaying near real-time pupil movement of the user while wearing the VR headset.
stitching in a compilation of the one or more captured static images of the user without the VR headset and the captured real-time pupil images of the user captured by the one or more camera devices located internal of the VR headset such that the generated 2D image is superimposed proximate to the location the headset image was virtually removed from the user wearing the VR headset-such that another user can view a VR image of the at least one user having the superimposed captured real-time pupil images and movement such that the at least one user was not wearing the VR headset.
14. Claims 16-17 19 21 23-25, 27, 29, and 31-33 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-4 of U.S. Patent No. 11,145,124 B2 (patent 124). Although the claims at issue are not identical, they are not patentably distinct from each other because while the claims are not the same, the claims of the present application are broader.
15. Claims 18, 26, and 35 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1 of patent 124 in view of Natsume. Natsume discloses in paragraph [0076] the storing of image compilation data into an external storage device such as a server, which can correspond to a data source. Claims 20 and 28 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1 of patent 124 in view of Malzbender (US Patent Application Publication No. 2012/0026277 A1) which discloses depth sensors. Claims 22 and 30 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1 of patent 124 in view of Arsenich (US Patent Application Publication No. 2009/0102915 A1) which discloses eye tracing micro video array sensor. Claim 34 is rejected on the ground of nonstatutory double patenting as being unpatentable over claim 1 of patent 124 in view of Deng (US Patent Application Publication No. 2016/0134803 A1) which discloses camera integrated into smartphone for capturing head/face. Although the claims at issue are not identical, they are not patentably distinct from each other because while the claims are not the same, the claims of present application are broader.
16. The following table shows correspondence between claims of present application and claims of patent 124.
Claims of present application
16
17
18
19
20
21
22
23
24
25
Claims of patent 124
1
1
1 in view of Natsume
2
1 in view of Malzbender
3
1 in view of Arsenich
4
1
1
26
27
28
29
30
31
32
33
34
35
1 in view of Natsume
2
1 in view of Malzbender
3
1 in view of Arsenich
4
1
2
1 in view of Deng
1 in view of Natsume
17. The following table shows correspondence between claim 24 of present application and claim 1 of patent 124.
Claim 24 of present application
Claims 1 of patent 124
24. (Newly Added) A computer system for creating Virtual Reality (VR) interactions between two or more users wherein at least one user is wearing a VR headset, comprising: one or more memory devices configured to store instructions thereon that, when executed by one or more processors, cause the one or more processors to:
generate a two-dimensional (2D) image of the user wearing the VR headset using one or more camera devices located external of the VR headset;
track the pupil movement of the user while wearing a VR headset using one or more sensors positioned internal of the VR headset;
1. A Virtual Reality (VR) computer system, comprising:
a VR headset to be worn by at least one user;
a computer processor having instructions to:
one or more pupil LED sensors located internal of the VR headset configured and operative to capture and track pupil movement of the at least one user wearing the VR headset;
at least one camera device positioned external of the VR headset operative to capture:
1) video image of at least one user wearing the VR headset; and
2) and one or more static images of the user not wearing the VR headset;
one or more camera devices located internal of the VR headset configured and operative to capture and record real-time images of a user's pupils simultaneous with the capturing and tracking of the user's pupil movement by the one or more pupil LED sensors; and
receive from the one or camera devices the recorded real-time images of the user's pupils while wearing the VR headset;
receive from the one or more pupil LED sensors the tracked pupil movement of the user's pupils captured simultaneous with the recorded real-time images of the user's pupils while wearing the VR headset;
remove an image of the VR headset from the generated 2D image of the user wearing the headset;
utilizing the recorded real-time images of the user's pupils and the simultaneously captured tracked pupil movements of the user's pupils while wearing the VR headset, generate a two-dimensional (2D) image of the least one user such that an image of the VR headset is virtually removed from the image of the at least one user wearing the VR headset by
and stitch a compilation image of the user consisting of one or more captured static images of the user without wearing the VR headset, responsive to the user's tracked pupil movement, in the generated 2D image of the user, whereby the compilation image is superimposed proximate to the location the VR headset image was removed from the user whereby another user can view a VR image of the user while wearing the VR headset as if the user was not wearing the VR headset while displaying near real-time pupil movement of the user while wearing the VR headset.
stitching in a compilation of the one or more captured static images of the user without the VR headset and the captured real-time pupil images of the user captured by the one or more camera devices located internal of the VR headset such that the generated 2D image is superimposed proximate to the location the headset image was virtually removed from the user wearing the VR headset such that another user can view a VR image of the at least one user having the superimposed captured real-time pupil images and movement such that the at least one user was not wearing the VR headset.
Claim Rejections - 35 USC § 103
18. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
19. The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
20. Claims 16-17, 19-21, 23 are rejected under 35 U.S.C. 103 as being unpatentable over Grossinger et al. (US Patent Application Publication No. 2017/0243334 A1) in view of Malzbender et al. (US Patent Application Publication No. 2012/0026277 A1).
21. Regarding Claim 16 (Newly Added), Grossinger discloses A computer-implemented method (Abstract reciting “A method and a system for reconstructing obstructed face portions are provided herein. …”) for creating Virtual Reality (VR) interactions between two or more users wherein at least one user is wearing a VR headset, (see FIG. 3; paragraph [0026] reciting “FIG. 3 is a schematic block diagram illustrating a virtual reality environment according to some embodiments of the present invention. Here, a third participant (not shown) is viewing two other participants 308A and 306A who wear helmets such as VR headsets 304A and 302A in the real world 300 which also contain real objects 303 and 305. In the virtual reality view 300B presented to the third user, the two other participants 308B and 306B are presented without the helmets as these were seamlessly replaced with reconstructed face images.” FIG. 3 shows 3 users interacting with each other and looking at each other.) comprising: generating a two-dimensional (2D) image of the user wearing the VR headset (paragraph [0010] reciting “… obtaining in real time, real-time 3D data, being 3D data of said head, wherein said person wears said face-obstructing object; …” Person wearing face-obstructing object (headset) is obtained in real time. 3D image data includes 2D image.)
tracking the pupil movement of the user while wearing a VR headset using one or more sensors positioned internal of the VR headset; (paragraph [0027] reciting “FIG. 4 is a schematic diagram illustrating an aspect in accordance with some embodiments according to the present invention. A user helmet 404 such as a VR headset is shown with a plurality of sensors 402 along its inner side (attachable to the user's face). The sensors are configured to sense the user's gestures such as the eyes (e.g., pupil of the eye and eyelid movement) and a computer processor is configured to modify a base image of the face based on the ongoing gestures and present them in real time (or with as little latency possible). By way of illustration, sensors 402 may sense movements of the facial skin that are indicative of the user laughing or smiling. In response, the base image of his or her face will undergo a process of image processing applying the effect of the laugh or the smile such as stretching various portions. This process is being updated in real time. Consequently, other viewers will be presented with the modified smiling face image of the user which will be seamlessly inserted into the image of the user, instead of the obstructed portion of the face.” Sensors 402 inside the user helmet 404 tracks the eye gestures in as near real time as possible.) removing an image of the VR headset from the generated 2D image of the user wearing the headset; (paragraph [0055] reciting “According to some embodiments of the present invention, the obstructed portions of the at least one person may be at least partially reconstructed based on occurrences in the VR environment. It is understood that this feature may also be implemented to the non-obstructed portions of the face.”)
and stitching a compilation image of the user consisting of one or more captured static images of the user without wearing the VR headset, (see FIG. 7 wherein the base image and combined with the 3D model into a reconstructed face image; paragraph [0047] reciting “FIG. 8 is a high level flowchart illustrating a method 800 according to some embodiments of the present invention. It is understood that method 800 may be implemented by an architecture that is different from the aforementioned architecture of system 500. Method 800 may include: obtaining off-line 3D data, being 3D data of a head of a person not wearing a face-obstructing object, being an object which obstructs a portion of the face of the person 810; obtaining in real time, real-time 3D data, being 3D data of said head, wherein said person wears said face-obstructing object 820; applying a 3D transformation to at least a portion of the off-line 3D data, based on the real-time 3D data, to yield reconstructed real time 3D data, being real-time 3D data related to the obstructed face portions 830; and merging the reconstructed real time 3D data into the real- time 3D data 840.” Reconstructed face is a compilation of user 3D image (which includes 2D static off-line images of the user head) with the user wearing the obstruction on the face in real time.) responsive to the user's tracked pupil movement, in the generated 2D image of the user, (paragraph [0043] reciting “The appearance of the eyes may be estimated based on another model, and using meta data 642 from sources external to real-time 2D data 622 such as which direction the person is looking at, given that in the virtual reality environment the person is interacting with another person (whose position is also known). Finally, reconstructed real-time 3D data (or model) 650 may be merged into real-time 3D data (or model) possibly with the obstructed portions segmented out 632, to yield a reconstructed real-time 3D data (or model) of the head of the person, with the obstructed portions reconstructed. Model 660 may then be generated into a 3D image to be presented to the other person(s) participating in the virtual reality environment.”) whereby the compilation image is superimposed proximate to the location the VR headset image was removed from the user whereby another user can view a VR image of the user while wearing the VR headset (paragraph [0043] reciting “… Finally, reconstructed real-time 3D data (or model) 650 may be merged into real-time 3D data (or model) possibly with the obstructed portions segmented out 632, to yield a reconstructed real-time 3D data (or model) of the head of the person, with the obstructed portions reconstructed. Model 660 may then be generated into a 3D image to be presented to the other person(s) participating in the virtual reality environment.” Real-time 3D reconstructed model of head corresponds to a compilation image.) as if the user was not wearing the VR headset (paragraph [0025] reciting “… A scene 200 in a virtual reality system is depicted, in which both persons (participants)
206 and 208 are wearing a near eye display or a VR headset 202 and 204 respectively, which happen to obstruct their faces. However, as opposed to FIG. 1, their respective views 210 and 212 which as presented via their near eye displays 202 and 204 show their counter participant not wearing the near eye display (real objects 203 and 205 are also not shown as the background of the real world is being replaced with a computer simulated environment). This is enabled by replacing part of the image that includes the obstructing object with a supplementary or modified face images that are being updated in real time, based on actual movements and/or expressions and/or gestures of the participants as will be explained in detail below.“) while displaying near real-time pupil movement of the user while wearing the VR headset. (paragraph [0043] reciting “… The appearance of the eyes may be estimated based on another model, and using meta data 642 from sources external to real-time 2D data 622 such as which direction the person is looking at, given that in the virtual reality environment the person is interacting with another person (whose position is also known). …” The eye positions are estimated in near real-time.)
While not explicitly disclosed by Grossinger, Malzbender discloses using one or more camera devices located external of the VR headset; (paragraph [0014] reciting “The video conference application 110 is an application that can be utilized in conjunction with one or more depth cameras 130 to determine distances of objects captured by one or more depth cameras 130 by determining distances of pixels that correspond to the objects. The pixels that correspond to the objects correspond to points on the objects. Additionally, the video conference application 110 controls one or more depth cameras 130 and/or one or more sensors 140 in identifying a head or face portion of a participant so as to align a region of interest on the participant and crop the region of interest from a captured video stream of the participant. Further, the video conference application 110 arranges the cropped video stream of the participant with additional cropped video streams of participants for displaying a video conference on the display device 150.”
It would have been obvious to a person of ordinary skills to use video cameras placed externally from the head to capture head images because Grossinger requires the depth sensors to capture one’s head wearing an obstruction.)
It would have been obvious to a person of ordinary skills in the art before the effective filing date of the claimed invention to modify Grossinger with Malzbender so that the video conferencing cameras can be used to capture the head of the user wearing the headset during virtual reality (VR) experience. This is an obvious modification since Grossinger already discloses using sensors to capture the face (Grossinger at FIG 5) but not explicitly video cameras. Thus, modifying the sensors in Grossingers with cameras achieves these goals.
22. Regarding Claim 17 (Newly Added), Grossinger further discloses The computer-implemented method as recited in claim 16, further including the step of generating the compilation image (see FIG. 7 wherein the reconstructed face image correspond to the compilation image.)
Malzbender further discloses by the one or more camera devices. (paragraph [0014] reciting “The video conference application 110 is an application that can be utilized in conjunction with one or more depth cameras 130 to determine distances of objects captured by one or more depth cameras 130 by determining distances of pixels that correspond to the objects. The pixels that correspond to the objects correspond to points on the objects. Additionally, the video conference application 110 controls one or more depth cameras 130 and/or one or more sensors 140 in identifying a head or face portion of a participant so as to align a region of interest on the participant and crop the region of interest from a captured video stream of the participant. Further, the video conference application 110 arranges the cropped video stream of the participant with additional cropped video streams of participants for displaying a video conference on the display device 150.”
It would have been obvious to a person of ordinary skills to use video cameras placed externally from the head to capture head images because Grossinger requires the depth sensors to capture one’s head wearing an obstruction.)
23. Regarding Claim 19 (Newly Added), Malzbender further discloses The computer-implemented method as recited in claim 16, wherein the 2D image is created using a video camera. (paragraph [0014] reciting “The video conference application 110 is an application that can be utilized in conjunction with one or more depth cameras 130 to determine distances of objects captured by one or more depth cameras 130 by determining distances of pixels that correspond to the objects. The pixels that correspond to the objects correspond to points on the objects. Additionally, the video conference application 110 controls one or more depth cameras 130 and/or one or more sensors 140 in identifying a head or face portion of a participant so as to align a region of interest on the participant and crop the region of interest from a captured video stream of the participant. Further, the video conference application 110 arranges the cropped video stream of the participant with additional cropped video streams of participants for displaying a video conference on the display device 150.”
It would have been obvious to a person of ordinary skills to use video cameras placed externally from the head to capture head images because Grossinger requires the depth sensors to capture one’s head wearing an obstruction.)
24. Regarding Claim 20 (Newly Added), Malzbender further discloses The computer-implemented method as recited in claim 16, wherein the one or more sensors includes a depth sensor. (paragraph [0004] reciting “FIG. 2 illustrates a depth camera capturing a video stream of a participant and detecting distances of pixels corresponding to objects within a viewing area of the depth camera according to an embodiment of the invention.” Would have been obvious to use depth cameras to capture eye gesture to generate realistic eye movement to be viewed in the virtual world.)
25. Regarding Claim 21. (Newly Added), Malzbender further discloses The computer-implemented method as recited in claim 16, wherein the one or more sensors include infrared LED sensors. (paragraph [0032] reciting “In one embodiment, one or more sensors 140 are infrared devices configured to identify a location of the participant's eyes. Utilizing the data from the location of the participant's eyes, one or more sensors 140 and the video conference application 110 can determine the location of the participant's face on the pixel map 190, binary map 195 and/or the video stream of the participant. In another embodiment, one or more sensors 140 are additional cameras configured to detect the participant's face utilizing face recognition technology. In other embodiments, the face recognition technology is a software application. Further, in additional embodiments, one or more sensors 140 are proximity sensors configured to detect proximities of one or more objects and/or one or more participants.” This is obvious since the one or more sensors are used to find the eyes.)
26. Regarding Claim 23 (Newly Added), Grossinger further discloses The method-implemented method as recited in claim 16, wherein user's may view each other in real-time. (see FIG. 2 wherein the users 208A and 26A are viewing each other in real time in the virtual world.)
27. Regarding Claim 24 (Newly Added), Grossinger discloses A computer system for creating Virtual Reality (VR) interactions (paragraph [0010] reciting “Some embodiments of the present invention overcome the aforementioned disadvantages of the prior art by providing a method and a system for reconstructing obstructed face portions for virtual reality environments. …”) between two or more users wherein at least one user is wearing a VR headset, (see FIG. 2 wherein both users are wearing headsets and having a virtual interaction in a virtual world.)
comprising: when executed by one or more processors, cause the one or more processors to: (paragraph [0010] reciting “… The system may implement the aforementioned steps over a computer processor.”) generate a two-dimensional (2D) image of the user wearing the VR headset (paragraph [0010] reciting “… obtaining in real time, real-time 3D data, being 3D data of said head, wherein said person wears said face-obstructing object; …” Person wearing face-obstructing object (headset) is obtained in real time. 3D image data includes 2D image.)
track the pupil movement of the user while wearing a VR headset using one or more sensors positioned internal of the VR headset; (paragraph [0027] reciting “FIG. 4 is a schematic diagram illustrating an aspect in accordance with some embodiments according to the present invention. A user helmet 404 such as a VR headset is shown with a plurality of sensors 402 along its inner side (attachable to the user's face). The sensors are configured to sense the user's gestures such as the eyes (e.g., pupil of the eye and eyelid movement) and a computer processor is configured to modify a base image of the face based on the ongoing gestures and present them in real time (or with as little latency possible). By way of illustration, sensors 402 may sense movements of the facial skin that are indicative of the user laughing or smiling. In response, the base image of his or her face will undergo a process of image processing applying the effect of the laugh or the smile such as stretching various portions. This process is being updated in real time. Consequently, other viewers will be presented with the modified smiling face image of the user which will be seamlessly inserted into the image of the user, instead of the obstructed portion of the face.” Sensors 402 inside the user helmet 404 tracks the eye gestures in as near real time as possible.) remove an image of the VR headset from the generated 2D image of the user wearing the headset; (paragraph [0055] reciting “According to some embodiments of the present invention, the obstructed portions of the at least one person may be at least partially reconstructed based on occurrences in the VR environment. It is understood that this feature may also be implemented to the non-obstructed portions of the face.”)
and stitch a compilation image of the user consisting of one or more captured static images of the user without wearing the VR headset, (see FIG. 7 wherein the base image and combined with the 3D model into a reconstructed face image; paragraph [0047] reciting “FIG. 8 is a high level flowchart illustrating a method 800 according to some embodiments of the present invention. It is understood that method 800 may be implemented by an architecture that is different from the aforementioned architecture of system 500. Method 800 may include: obtaining off-line 3D data, being 3D data of a head of a person not wearing a face-obstructing object, being an object which obstructs a portion of the face of the person 810; obtaining in real time, real-time 3D data, being 3D data of said head, wherein said person wears said face-obstructing object 820; applying a 3D transformation to at least a portion of the off-line 3D data, based on the real-time 3D data, to yield reconstructed real time 3D data, being real-time 3D data related to the obstructed face portions 830; and merging the reconstructed real time 3D data into the real- time 3D data 840.” Reconstructed face is a compilation of user 3D image (which includes 2D static off-line images of the user head) with the user wearing the obstruction on the face in real time.) responsive to the user's tracked pupil movement, in the generated 2D image of the user, (paragraph [0043] reciting “The appearance of the eyes may be estimated based on another model, and using meta data 642 from sources external to real-time 2D data 622 such as which direction the person is looking at, given that in the virtual reality environment the person is interacting with another person (whose position is also known). Finally, reconstructed real-time 3D data (or model) 650 may be merged into real-time 3D data (or model) possibly with the obstructed portions segmented out 632, to yield a reconstructed real-time 3D data (or model) of the head of the person, with the obstructed portions reconstructed. Model 660 may then be generated into a 3D image to be presented to the other person(s) participating in the virtual reality environment.”) whereby the compilation image is superimposed proximate to the location the VR headset image was removed from the user whereby another user can view a VR image of the user while wearing the VR headset (paragraph [0043] reciting “… Finally, reconstructed real-time 3D data (or model) 650 may be merged into real-time 3D data (or model) possibly with the obstructed portions segmented out 632, to yield a reconstructed real-time 3D data (or model) of the head of the person, with the obstructed portions reconstructed. Model 660 may then be generated into a 3D image to be presented to the other person(s) participating in the virtual reality environment.” Real-time 3D reconstructed model of head corresponds to a compilation image.) as if the user was not wearing the VR headset (paragraph [0025] reciting “… A scene 200 in a virtual reality system is depicted, in which both persons (participants)
206 and 208 are wearing a near eye display or a VR headset 202 and 204 respectively, which happen to obstruct their faces. However, as opposed to FIG. 1, their respective views 210 and 212 which as presented via their near eye displays 202 and 204 show their counter participant not wearing the near eye display (real objects 203 and 205 are also not shown as the background of the real world is being replaced with a computer simulated environment). This is enabled by replacing part of the image that includes the obstructing object with a supplementary or modified face images that are being updated in real time, based on actual movements and/or expressions and/or gestures of the participants as will be explained in detail below.“) while displaying near real-time pupil movement of the user while wearing the VR headset. (paragraph [0043] reciting “… The appearance of the eyes may be estimated based on another model, and using meta data 642 from sources external to real-time 2D data 622 such as which direction the person is looking at, given that in the virtual reality environment the person is interacting with another person (whose position is also known). …” The eye positions are estimated in near real-time.)
While Grossinger does not explicitly disclose, Malzbender discloses one or more memory devices configured to store instructions thereon that, (paragraph [0063] reciting “… As noted above, in one embodiment, the Video Conference Application
610 is firmware that is embedded into one or more components of the machine 600 as ROM. In other embodiments, the Video Conference Application 610 is a software application which is stored and accessed from a hard drive, a compact disc, a flash disk, a network drive or any other form o computer readable medium that is coupled to the machine 600.”)
using one or more camera devices located external of the VR headset; (paragraph [0014] reciting “The video conference application 110 is an application that can be utilized in conjunction with one or more depth cameras 130 to determine distances of objects captured by one or more depth cameras 130 by determining distances of pixels that correspond to the objects. The pixels that correspond to the objects correspond to points on the objects. Additionally, the video conference application 110 controls one or more depth cameras 130 and/or one or more sensors 140 in identifying a head or face portion of a participant so as to align a region of interest on the participant and crop the region of interest from a captured video stream of the participant. Further, the video conference application 110 arranges the cropped video stream of the participant with additional cropped video streams of participants for displaying a video conference on the display device 150.”
It would have been obvious to a person of ordinary skills to use video cameras placed externally from the head to capture head images because Grossinger requires the depth sensors to capture one’s head wearing an obstruction.)
It would have been obvious to a person of ordinary skills in the art before the effective filing date of the claimed invention to modify Grossinger with Malzbender so that the video conferencing cameras can be used to capture the head of the user wearing the headset during virtual reality (VR) experience. This is an obvious modification since Grossinger already discloses using sensors to capture the face (Grossinger at FIG 5) but not explicitly video cameras. Thus, modifying the sensors in Grossingers with cameras achieves these goals.
28. Regarding Claim 25 (Newly Added), Grossinger further discloses The computer system as recited in claim 24, wherein the processor is further configured to generate the compilation image (see FIG. 7 wherein the reconstructed face image correspond to the compilation image.)
Malzbender further discloses by the one or more camera devices. (paragraph [0014] reciting “The video conference application 110 is an application that can be utilized in conjunction with one or more depth cameras 130 to determine distances of objects captured by one or more depth cameras 130 by determining distances of pixels that correspond to the objects. The pixels that correspond to the objects correspond to points on the objects. Additionally, the video conference application 110 controls one or more depth cameras 130 and/or one or more sensors 140 in identifying a head or face portion of a participant so as to align a region of interest on the participant and crop the region of interest from a captured video stream of the participant. Further, the video conference application 110 arranges the cropped video stream of the participant with additional cropped video streams of participants for displaying a video conference on the display device 150.”
It would have been obvious to a person of ordinary skills to use video cameras placed externally from the head to capture head images because Grossinger requires the depth sensors to capture one’s head wearing an obstruction.)
29. Regarding Claim 27 (Newly Added), Malzbender further discloses The computer system as recited in claim 24, wherein the 2D image is created using a video camera. (paragraph [0014] reciting “The video conference application 110 is an application that can be utilized in conjunction with one or more depth cameras 130 to determine distances of objects captured by one or more depth cameras 130 by determining distances of pixels that correspond to the objects. The pixels that correspond to the objects correspond to points on the objects. Additionally, the video conference application 110 controls one or more depth cameras 130 and/or one or more sensors 140 in identifying a head or face portion of a participant so as to align a region of interest on the participant and crop the region of interest from a captured video stream of the participant. Further, the video conference application 110 arranges the cropped video stream of the participant with additional cropped video streams of participants for displaying a video conference on the display device 150.”
It would have been obvious to a person of ordinary skills to use video cameras placed externally from the head to capture head images because Grossinger requires the depth sensors to capture one’s head wearing an obstruction.)
30. Regarding Claim 28 (Newly Added), Malzbender further discloses The computer system as recited in claim 24, wherein the one or more sensors includes a depth sensor. (paragraph [0004] reciting “FIG. 2 illustrates a depth camera capturing a video stream of a participant and detecting distances of pixels corresponding to objects within a viewing area of the depth camera according to an embodiment of the invention.” Would have been obvious to use depth cameras to capture eye gesture to generate realistic eye movement to be viewed in the virtual world.)
31. Regarding Claim 29 (Newly Added), Malzbender further discloses The computer system as recited in claim 24, wherein the one or more sensors include infrared LED sensors. (paragraph [0032] reciting “In one embodiment, one or more sensors 140 are infrared devices configured to identify a location of the participant's eyes. Utilizing the data from the location of the participant's eyes, one or more sensors 140 and the video conference application 110 can determine the location of the participant's face on the pixel map 190, binary map 195 and/or the video stream of the participant. In another embodiment, one or more sensors 140 are additional cameras configured to detect the participant's face utilizing face recognition technology. In other embodiments, the face recognition technology is a software application. Further, in additional embodiments, one or more sensors 140 are proximity sensors configured to detect proximities of one or more objects and/or one or more participants.” This is obvious since the one or more sensors are used to find the eyes.)
32. Regarding Claim 31 (Newly Added), Grossinger further discloses The computer system as recited in claim 24, wherein user's may view each other in real-time. (see FIG. 2 wherein the users 208A and 26A are viewing each other in real time in the virtual world.)
33. Regarding Claim 32 (Newly Added), Grossinger discloses A Virtual Reality (VR) system for creating VR interactions (paragraph [0010] reciting “Some embodiments of the present invention overcome the aforementioned disadvantages of the prior art by providing a method and a system for reconstructing obstructed face portions for virtual reality environments. …”) between two or more users wherein at least one user is wearing a VR headset, (see FIG. 2 wherein both users are wearing headsets and having a virtual interaction in a virtual world.) comprising:
at least one VR headset device having one or more integrated sensor components located internal of the VR headset; (paragraph [0027] reciting “FIG. 4 is a schematic diagram illustrating an aspect in accordance with some embodiments according to the present invention. A user helmet 404 such as a VR headset is shown with a plurality of sensors 402 along its inner side (attachable to the user's face). The sensors are configured to sense the user's gestures such as the eyes (e.g., pupil of the eye and eyelid movement) and a computer processor is configured to modify a base image of the face based on the ongoing gestures and present them in real time (or with as little latency possible). By way of illustration, sensors 402 may sense movements of the facial skin that are indicative of the user laughing or smiling. In response, the base image of his or her face will undergo a process of image processing applying the effect of the laugh or the smile such as stretching various portions. This process is being updated in real time. Consequently, other viewers will be presented with the modified smiling face image of the user which will be seamlessly inserted into the image of the user, instead of the obstructed portion of the face.” Sensors 402 inside the user helmet 404 tracks the eye gestures in as near real time as possible.)
an image of a user wearing the VR headset set device (see FIG. 5 wherein face capturing 530 captures face wearing obstruction device. paragraph [0010] reciting “… obtaining in real time, real-time 3D data, being 3D data of said head, wherein said person wears said face-obstructing object; …” Person wearing face-obstructing object (headset) is obtained in real time. 3D image data includes 2D image.) a VR imaging computer device communicatively coupled to each of the at least one VR headset device and camera device, (see FIG. 5, FIG. 6 and FIG. 7 wherein the systems 500, 600, and 700 correspond to a VR imaging computer device that is couple to every worn headset device and removes them from VR interaction) and a processor disposed in communication (paragraph [0010] reciting “… The system may implement the aforementioned steps over a computer processor.”)
generate a two-dimensional (2D) image of the user wearing the VR headset using the camera device; (paragraph [0010] reciting “… obtaining in real time, real-time 3D data, being 3D data of said head, wherein said person wears said face-obstructing object; …” Person wearing face-obstructing object (headset) is obtained in real time. 3D image data includes 2D image.)
track the pupil movement of the user while wearing the VR headset using the one or more sensor components; (paragraph [0027] reciting “FIG. 4 is a schematic diagram illustrating an aspect in accordance with some embodiments according to the present invention. A user helmet 404 such as a VR headset is shown with a plurality of sensors 402 along its inner side (attachable to the user's face). The sensors are configured to sense the user's gestures such as the eyes (e.g., pupil of the eye and eyelid movement) and a computer processor is configured to modify a base image of the face based on the ongoing gestures and present them in real time (or with as little latency possible). By way of illustration, sensors 402 may sense movements of the facial skin that are indicative of the user laughing or smiling. In response, the base image of his or her face will undergo a process of image processing applying the effect of the laugh or the smile such as stretching various portions. This process is being updated in real time. Consequently, other viewers will be presented with the modified smiling face image of the user which will be seamlessly inserted into the image of the user, instead of the obstructed portion of the face.” Sensors 402 inside the user helmet 404 tracks the eye gestures in as near real time as possible.) remove an image of the VR headset from the generated 2D image of the user wearing the headset; (paragraph [0055] reciting “According to some embodiments of the present invention, the obstructed portions of the at least one person may be at least partially reconstructed based on occurrences in the VR environment. It is understood that this feature may also be implemented to the non-obstructed portions of the face.”)
and stitch a compilation image of the user consisting of one or more captured static images of the user without wearing the VR headset, (see FIG. 7 wherein the base image and combined with the 3D model into a reconstructed face image; paragraph [0047] reciting “FIG. 8 is a high level flowchart illustrating a method 800 according to some embodiments of the present invention. It is understood that method 800 may be implemented by an architecture that is different from the aforementioned architecture of system 500. Method 800 may include: obtaining off-line 3D data, being 3D data of a head of a person not wearing a face-obstructing object, being an object which obstructs a portion of the face of the person 810; obtaining in real time, real-time 3D data, being 3D data of said head, wherein said person wears said face-obstructing object 820; applying a 3D transformation to at least a portion of the off-line 3D data, based on the real-time 3D data, to yield reconstructed real time 3D data, being real-time 3D data related to the obstructed face portions 830; and merging the reconstructed real time 3D data into the real- time 3D data 840.” Reconstructed face is a compilation of user 3D image (which includes 2D static off-line images of the user head) with the user wearing the obstruction on the face in real time.) responsive to the user's tracked pupil movement, in the generated 2D image of the user, (paragraph [0043] reciting “The appearance of the eyes may be estimated based on another model, and using meta data 642 from sources external to real-time 2D data 622 such as which direction the person is looking at, given that in the virtual reality environment the person is interacting with another person (whose position is also known). Finally, reconstructed real-time 3D data (or model) 650 may be merged into real-time 3D data (or model) possibly with the obstructed portions segmented out 632, to yield a reconstructed real-time 3D data (or model) of the head of the person, with the obstructed portions reconstructed. Model 660 may then be generated into a 3D image to be presented to the other person(s) participating in the virtual reality environment.”) whereby the compilation image is superimposed proximate to the location the VR headset image was removed from the user whereby another user can view a VR image of the user while wearing the VR headset (paragraph [0043] reciting “… Finally, reconstructed real-time 3D data (or model) 650 may be merged into real-time 3D data (or model) possibly with the obstructed portions segmented out 632, to yield a reconstructed real-time 3D data (or model) of the head of the person, with the obstructed portions reconstructed. Model 660 may then be generated into a 3D image to be presented to the other person(s) participating in the virtual reality environment.” Real-time 3D reconstructed model of head corresponds to a compilation image.) as if the user was not wearing the VR headset (paragraph [0025] reciting “… A scene 200 in a virtual reality system is depicted, in which both persons (participants)
206 and 208 are wearing a near eye display or a VR headset 202 and 204 respectively, which happen to obstruct their faces. However, as opposed to FIG. 1, their respective views 210 and 212 which as presented via their near eye displays 202 and 204 show their counter participant not wearing the near eye display (real objects 203 and 205 are also not shown as the background of the real world is being replaced with a computer simulated environment). This is enabled by replacing part of the image that includes the obstructing object with a supplementary or modified face images that are being updated in real time, based on actual movements and/or expressions and/or gestures of the participants as will be explained in detail below.“) while displaying near real-time pupil movement of the user while wearing the VR headset. (paragraph [0043] reciting “… The appearance of the eyes may be estimated based on another model, and using meta data 642 from sources external to real-time 2D data 622 such as which direction the person is looking at, given that in the virtual reality environment the person is interacting with another person (whose position is also known). …” The eye positions are estimated in near real-time.)
While Grossinger does not explicitly disclose, Malzbender discloses a camera device for at least capturing located external of the VR headset; (paragraph [0014] reciting “The video conference application 110 is an application that can be utilized in conjunction with one or more depth cameras 130 to determine distances of objects captured by one or more depth cameras 130 by determining distances of pixels that correspond to the objects. The pixels that correspond to the objects correspond to points on the objects. Additionally, the video conference application 110 controls one or more depth cameras 130 and/or one or more sensors 140 in identifying a head or face portion of a participant so as to align a region of interest on the participant and crop the region of interest from a captured video stream of the participant. Further, the video conference application 110 arranges the cropped video stream of the participant with additional cropped video streams of participants for displaying a video conference on the display device 150.”
It would have been obvious to a person of ordinary skills to use video cameras placed externally from the head to capture head images because Grossinger requires the depth sensors to capture one’s head wearing an obstruction.)
the VR imaging computer device including a memory with said memory and storage device, and configured to issue a plurality of instructions stored in the memory wherein the instructions cause the processor to: (paragraph [0063] reciting “… As noted above, in one embodiment, the Video Conference Application
610 is firmware that is embedded into one or more components of the machine 600 as ROM. In other embodiments, the Video Conference Application 610 is a software application which is stored and accessed from a hard drive, a compact disc, a flash disk, a network drive or any other form o computer readable medium that is coupled to the machine 600.”)
It would have been obvious to a person of ordinary skills in the art before the effective filing date of the claimed invention to modify Grossinger with Malzbender so that the video conferencing cameras can be used to capture the head of the user wearing the headset during virtual reality (VR) experience. This is an obvious modification since Grossinger already discloses using sensors to capture the face (Grossinger at FIG 5) but not explicitly video cameras. Thus, modifying the sensors in Grossingers with cameras achieves these goals.
34. Regarding Claim 33 (Newly Added), Malzbender further discloses The VR system as recited in claim 32, wherein the camera device is a video camera. (paragraph [0014] reciting “The video conference application 110 is an application that can be utilized in conjunction with one or more depth cameras 130 to determine distances of objects captured by one or more depth cameras 130 by determining distances of pixels that correspond to the objects. The pixels that correspond to the objects correspond to points on the objects. Additionally, the video conference application 110 controls one or more depth cameras 130 and/or one or more sensors 140 in identifying a head or face portion of a participant so as to align a region of interest on the participant and crop the region of interest from a captured video stream of the participant. Further, the video conference application 110 arranges the cropped video stream of the participant with additional cropped video streams of participants for displaying a video conference on the display device 150.”
It would have been obvious to a person of ordinary skills to use video cameras placed externally from the head to capture head images because Grossinger requires the depth sensors to capture one’s head wearing an obstruction.)
35. Claims 18, 26, and 35 are rejected under 35 U.S.C. 103 as being unpatentable over Grossinger in view of Malzbender and further in view of Natsume.
36. Regarding Claim 18 (Newly Added), while the combination of Grossinger and Malzbender does not explicitly disclose, Natsume discloses The computer-implemented method as recited in claim 17, further including the step of storing the compilation image in a data source. (paragraph [0076] reciting “… The image data transmitting unit 19 transmits either converted image data and image compilation data to the receiving apparatus 30 or converted image data to an external storage destination (a server) 50. …”)
It would have been obvious to a person of ordinary skills in the art before the effective filing date of the claimed invention to modify Grossinger and Malzbender with Natsume so that the reconstructed 3D facial model is stored for future retrieval. This allows for to see how he or she looks like in the virtual world.
37. Regarding Claim 26 (Newly Added), while the combination of Grossinger and Malzbender does not explicitly disclose, Natsume discloses The computer system as recited in claim 25, wherein the processor is further configured to store the compilation image in a data source. (paragraph [0076] reciting “… The image data transmitting unit 19 transmits either converted image data and image compilation data to the receiving apparatus 30 or converted image data to an external storage destination (a server) 50. …”)
It would have been obvious to a person of ordinary skills in the art before the effective filing date of the claimed invention to modify Grossinger and Malzbender with Natsume so that the reconstructed 3D facial model is stored for future retrieval. This allows for to see how he or she looks like in the virtual world.
38. Regarding Claim 35 (Newly Added), while the combination of Grossinger and Malzbender does not explicitly disclose, Natsume discloses The VR system as recited in claim 32, wherein the processor is further configured to generate the compilation image by the one or more camera devices and store the compilation image in a data source. (paragraph [0076] reciting “… The image data transmitting unit 19 transmits either converted image data and image compilation data to the receiving apparatus 30 or converted image data to an external storage destination (a server) 50. …”)
It would have been obvious to a person of ordinary skills in the art before the effective filing date of the claimed invention to modify Grossinger and Malzbender with Natsume so that the reconstructed 3D facial model is stored for future retrieval. This allows for to see how he or she looks like in the virtual world.
39. Claims 22 and 30 are rejected under 35 U.S.C. 103 as being unpatentable over Grossinger in view of Malzbender and further in view of Sryatoslav Ivanovich Arsenich (US Patent Application Publication No. 2009/0102915 A1).
40. Regarding Claim 22 (Newly Added), while the combination of Grossinger and Malzbender does not explicitly disclose, Arsenich discloses The computer-implemented method as recited in claim 21, wherein the one or more sensors further include micro video camera for recording aspects of a user's pupil. (paragraph [0082] reciting “FIGS. 12 and 13 show a head-mounted stereoprojector with a reflecting spherical stereoscreen in the form of the mirror glasses. The system is fixed on the head by means of an elastic rim or a strip 28. On the forehead in front of the stereoscreen two microprojectors are fixed: 25 l—the projector forming left frame and 25 r—the projector forming right frame. The projectors comprise the movable microprojection units 8 mounted on the auto-drives 9. The optical systems of projection magnification 26 l and 26 r are intended for forming of point focal zones of stereo viewing of left and right frames of the stereopair (focused by the stereoscreen 1 in the pupils of the corresponding eyes of the viewer). On the stereoscopic glasses the micro video cameras 4 r and 4 l are mounted with the tracking system 3 for monitoring of the pupils of the eyes, connected with the auto-correctors 10 and video corrector 11. The stereoscreen 1 is movable and mounted on the auto-drive 2, which allows performing of auto-correction.”)
It would have been obvious to a person of ordinary skills in the art before the effective filing date of the claimed invention to modify Grossinger and Malzbender with Arsenich so that micro video cameras as used monitor eye movement within the headset. This is an obviously beneficial modification since Grossinger discloses the requirement to sense eye gesture using sensors and micro video cameras can be the sensors that captures the images of the eyes for real time displaying during VR sessions.
41. Regarding Claim 30 (Newly Added), while the combination of Grossinger and Malzbender does not explicitly disclose, Arsenich discloses The computer system as recited in claim 29, wherein the one or more sensors further include micro video camera for recording aspects of a user's pupil. (paragraph [0082] reciting “FIGS. 12 and 13 show a head-mounted stereoprojector with a reflecting spherical stereoscreen in the form of the mirror glasses. The system is fixed on the head by means of an elastic rim or a strip 28. On the forehead in front of the stereoscreen two microprojectors are fixed: 25 l—the projector forming left frame and 25 r—the projector forming right frame. The projectors comprise the movable microprojection units 8 mounted on the auto-drives 9. The optical systems of projection magnification 26 l and 26 r are intended for forming of point focal zones of stereo viewing of left and right frames of the stereopair (focused by the stereoscreen 1 in the pupils of the corresponding eyes of the viewer). On the stereoscopic glasses the micro video cameras 4 r and 4 l are mounted with the tracking system 3 for monitoring of the pupils of the eyes, connected with the auto-correctors 10 and video corrector 11. The stereoscreen 1 is movable and mounted on the auto-drive 2, which allows performing of auto-correction.”)
It would have been obvious to a person of ordinary skills in the art before the effective filing date of the claimed invention to modify Grossinger and Malzbender with Arsenich so that micro video cameras as used monitor eye movement within the headset. This is an obviously beneficial modification since Grossinger discloses the requirement to sense eye gesture using sensors and micro video cameras can be the sensors that captures the images of the eyes for real time displaying during VR sessions.
42. Claim 34 is rejected under 35 U.S.C. 103 as being unpatentable over Grossinger in view of Malzbender and further in view of Yining Deng (US Patent Application Publication No. 2016/0134803 A1).
43. Regarding Claim 34 (Newly Added), while the combination of Grossinger and Malzbender does not explicitly disclose, Deng discloses The VR system as recited in claim 33, camera devices is intergrated with a smart phone device. (paragraph [0024] reciting “… Additionally or alternately, indicators 309 that inform a user how to move the smartphone relative to their head or that act as warnings when a user's head is out of position may help a user position the camera appropriately to a capture images at particular perspective angles. Indicators may include visual cues, such as arrows presented on the display of a smartphone, according to some embodiments. Audible cues may also be used to help a user position the camera to capture an image at a particular angle.”)
It would have been obvious to a person of ordinary skills in the art before the effective filing date of the claimed invention to modify Grossinger and Malzbender with Deng so that a smartphone integrated camera can be used to capture head the of the user. This is a beneficial modification since Grossinger at FIG. 5 discloses using sensors to capture the face of the user and smartphone cameras are easy and reliable camera to make such facial image captures.
CONTACT
Any inquiry concerning this communication or earlier communications from the examiner should be directed to FRANK S CHEN whose telephone number is (571)270-7993. The examiner can normally be reached Mon - Fri 8-11:30 and 1:30-6.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee Tung can be reached at 5712727794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/FRANK S CHEN/Primary Examiner, Art Unit 2611