DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 1-15 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
Claims 1-15 recite “obtain a distance between the display and an object located in an expected field of view of user observable by the user if the user were not wearing the head mounted display system;” however, Specification does not describe this feature. For example, Specification, Paragraph 18 determines distance of the object to the head mounted display unit (HMD unit) or the depth sensor of the HMD unit, but not to the display of the HMD unit as claimed.
Claims 2 and 14 are similarly rejected for reasons stated above.
Claims 1-15 further recite: “determine, based on the location of each of the plurality of video cameras relative to the display, their fixed respective fields of view and the distance between the display and the object, specific video cameras of the plurality of video cameras to capture light rays” however Specification does not recite this feature. According to Specification, Paragraphs 33, 36, “depth information 160, from depth sensors 110 if present, is captured … computational device 112 uses the depth information 160 with the light field rendering technique to estimate the specific cameras 124a, 124b, etc.,” which does not mention location of cameras, fields of view, or distance between the display and the object. It is also not clear as to what is being determined for the specific video cameras since Specification, Paragraph 7 also indicates “Specifically, the video cameras must be mounted at different physical locations than the pupils of the user's eyes and thus the captured video images which are displayed to the user on the head mounted display do not accurately correspond to the user's pupil position and distance of the user to the observed portion of the real world.” This means that multiple cameras can be “estimated” to capture the light rays directed at a virtual location of the user’s eyes, and at the same time that none of the cameras will do it accurately.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
For purposes of compact prosecution, Claims 1-15 are rejected under 35 U.S.C. 103 as being unpatentable over US 20180033209 to Akeley (“Akeley”) in view of US 20150206329 to Devries (“Devries”).
Regarding Claim 1: “A head mounted display system comprising: (“The viewer may wear a head-mounted display (HMD)” Akeley, Paragraph 91.)
a display capable of being worn by a user in front of their eyes and displaying images to the user; (“The viewer may wear a head-mounted display (HMD) such as the Oculus, which both tracks the viewer's head position and orientation, and facilitates the display of separately computed images to each eye” Akeley, Paragraph 91. See a similar embodiment in Devries, Paragraph 26.)
a plurality of video cameras having fixed respective fields of view relative to pupils of the eyes of the user when the head mounted display system is worn by the user, the plurality of video cameras operable to capture video images from the respective fields of view; … specific video cameras of the plurality of video cameras to capture light rays that would reach the pupils of the eyes of the user from the object if the user was observing the object without wearing the head mounted display system; (Under the broadest reasonable interpretation consistent with the specification and ordinary skill in the art, the plurality of video cameras either simulate the locations of the eyes of the user so as to capture video that can be displayed to the user using an HMD, or the cameras are physically located inside an HMD where the eyes of the user would be expected had the user worn it instead.
Prior art teaches the first embodiment: “The video data may be captured with a plurality of cameras, each attached to a capture rig such as a tiled camera array, with positions and orientations chosen such that the cameras' fields of view overlap within the desired capture field of view.” Akeley, Paragraph 91. This data is “such that the vantages can be used to generate viewpoint video of the scene, as viewed from at least two virtual viewpoints corresponding to viewpoints of an actual viewer's eyes within the viewing volume,” and thus fixed to the respective fields of view of the eyes by this mechanism. See Akeley, Paragraph 13 and similarly in Devries Paragraphs 26. See cumulative treatment of the second embodiment, of fixing cameras on the head mounted device below.)
a computational device operable to: … determine a location of each camera of the plurality of video cameras relative to the display (Since the cameras (above) are located to designate virtual or actual locations of the viewer’s eyes, determination of the locations of the cameras is the same process as determination of location of the user’s eyes. Prior art teaches this: “The viewer may wear a head-mounted display (HMD) such as the Oculus, which both tracks the viewer's head position and orientation, and facilitates the display of separately computed images to each eye … the images presented to the viewer's eyes are ideally correct for both the position and orientation of his eyes,” which thus tracks the location of the eye inside the display structure. See Akeley, Paragraph 92. Cumulatively, for the embodiment where the cameras and the eyes are in different or virtualized locations, prior art also teaches: “it may be necessary to compute the eye's image [at a known location in the HMD] from one or more camera images at position(s) and/or orientation(s) that are different from those of the eye.” Akeley, Paragraph 92.)
obtain a distance between the display and an object located in an expected field of view of user observable by the user if the user were not wearing the head mounted display system; (Under the broadest reasonable interpretation consistent with the specification and ordinary skill in the art, expected field of view of user observable by the user if the user were not wearing the head mounted display system does not affect the distance between the display and the object; here the object can be located in the expected camera field of view or a field of view that is displayed by the HMD. Prior art teaches variant embodiments. For example, “Depth information may also be computed for each pixel location in the camera-captured image. Certain digital cameras compute this information directly, for example by measuring the time of flight of photons from the scene object to the camera,” which also provides a distance to the display having the cameras or depth sensors or distance that is visible when the camera images are displayed at the HMD. See Akeley, Paragraph 72. Also note that images can be reprojected for the location of a particular vantage in the 3D space and/or for the location of the head and the corresponding head mounted display from any other vantage. Akeley, Paragraphs 18, 38, 42, 91-92. See treatment of cameras including depth-measuring cameras mounted to the display below.)
determine, based on the location of each of the plurality of video cameras relative to the display, their fixed respective fields of view and the distance between the display and the object, specific video cameras of the plurality of video cameras to capture light rays that would reach the pupils of the eyes of the user from the object if the user was observing the object without wearing the head mounted display system; (Again, this limitation is not clear. In the embodiment where the cameras mark the locations of the eyes in the HMD, Prior Art teaches: “For playback to be immersive, the images presented to the viewer's eyes are ideally correct for both the position and orientation of his eyes,” as marked by the respective video camera viewpoint that is “selectively delivered to the viewer based on the position and/or orientation of the viewer's head …” See Akeley, Paragraphs 92, 127. In another embodiment where the vantage of the video cameras is adapted to be viewed by the eyes of the user, Prior Art teaches: “to compute the eye's image from one or more camera images at position(s) and/or orientation(s) that are different from those of the eye.” See Akeley, Paragraph 92.)
determine based on the fixed respective fields of view of the specific video cameras, and the distance, a transformation to transform captured video images from each of the specific video cameras to the selected locations of the pupils of the user; and (“The viewer may wear a head-mounted display (HMD) such as the Oculus, which both tracks the viewer's head position and orientation, and facilitates the display of separately computed images to each eye at a high (e.g., 90 Hz) frame rate. … For playback to be immersive, the images presented to the viewer's eyes are ideally correct for both the position and orientation of his eyes. In general, the position and orientation of an eye will not match that of any camera, so it may be necessary to compute the eye's image [transform captured images] from one or more camera images at position(s) and/or orientation(s) that are different from those of the eye.” Akeley, Paragraphs 91-92. The eye images may be computed for selected vantages: “Each of the vantages may be an image computed from the camera images. The vantages may have positions that are distributed throughout a 3D viewing volume.” Akeley, Paragraphs 93-94.)
apply the transformation to the captured video images of the specific video cameras to generate an image for display on the display, the generated image corresponding to a virtual viewpoint at the pupils of the user, (Prior art teaches an embodiment within this scope: “the images presented to the viewer's eyes are ideally correct for both the position and orientation of his eyes. … it may be necessary to compute the eye's image from one or more camera images at position(s) and/or orientation(s) that are different from those of the eye.” Akeley, Paragraphs 92-94. See a similar computational device in Devries, Paragraph 26.
Under the broadest reasonable interpretation consistent with the specification and ordinary skill in the art, the images can inherently be generated for display in real time or not real time, i.e. at the speed of the image processing. The feature is rejected for reasons of inherency and cumulatively because prior art teaches: “The viewpoint video may be generated in real-time” Akelay, Paragraph 141. )
including displaying the objects in the respective fields of view of the specific video cameras at depths corresponding to distances of the object from the selected virtual locations of the pupils of the user, … the depths computed based on the obtained distance and the fixed respective fields of view of the specific video cameras relative to the selected virtual locations of the pupils of the user.“ (“In general, the position and orientation of an eye will not match that of any camera, so it may be necessary to compute the eye's image [transform captured images] from one or more camera images at position(s) and/or orientation(s) that are different from those of the eye,” where positions include camera distances to the object that are also transformed to display the object at a corresponding distance to the eyes. Akeley, Paragraphs 91-92, 76, and Fig. 2. As noted above, a user is free to select such virtual locations in the viewing volume.)
Akeley does not teach a relevant embodiment of “specific video cameras having fixed respective fields of view relative to pupils of the eyes of the user when the head mounted display system is worn by the user, the at least two video cameras operable to capture video images from the respective fields of view;” where the at least two video cameras are mounted adjacent the display, i.e. also head mounted.
Devries teaches the above claim feature in the context of a head mounted display: “computing system 208 may use data from, among other sources, various sensors and cameras ( e.g. outward facing camera that obtain digital images of object 204) to determine a displayed image that may be displayed to the wearer” Devries, Paragraph 26. See eye facing digital cameras 221 and object facing digital cameras 310 in Devries, Figs. 2-3. Also note that the cameras can be “in-depth cameras, stereo cameras [dual cameras], … and/or single 2D cameras.” Devries, Paragraph 40.
Therefore, before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to supplement the teachings of Akeley so that “at least two video cameras mounted adjacent the at least one display” as taught in Devries, in order to track user input or to determine images that can be displayed to the user directly or in an augmented manner. Devries, Paragraphs 26, 40.
Finally, in reviewing the present application, there does not seem to be objective evidence that the claim limitations are particularity directed to: addressing a particular problem which was recognized but unsolved in the art, producing unexpected results at the level of the ordinary skill in the art, or any other objective indicators of non-obviousness.
Regarding Claim 2: “The head mounted display system according to claim 1 wherein the computational device is operable to compute the distances between the display and the objects from the captured video images.” (“Depth information may also be computed for each pixel location in the camera-captured image. Certain digital cameras compute this information directly, for example by measuring the time of flight of photons from the scene object to the camera.” Akeley, Paragraph 72. Also note that this distance can be reprojected with respect to the location of they eye or the head mounted display. Akeley, Paragraphs 91-92, 76, and Fig. 2. Also note the depth-cameras and depth aware cameras mounted next to the display that directly capture distances from the display and the object in Devries, Paragraphs 40, 44 and Figs. 2-3. See statement of motivation in Claim 1.)
Regarding Claim 3: “The head mounted display system according to claim 2, wherein the computational device is operable to generate an image for each pupil of the user, each generated image corresponding to the viewpoint of the respective pupil of the user and each generated image is displayed to the respective pupil of the user providing the user with a stereoscopic image.” (“the techniques described herein can be used to generate stereo images … to provide different output video for each of the two eyes.” Akeley, Paragraphs 110-111.)
Regarding Claim 4: “The head mounted display system according to claim 2, wherein the selected virtual locations of the pupils of the user are selected by the user.” ( “a viewer can freely position his or her head to view the environment from the corresponding position and viewing direction. … The viewing experience is immersive, meaning that the viewer sees the environment from his or her position and orientation as though he or she were actually in the scene at that position and orientation,” and thus a different viewing location of the pupils. Akely, Paragraph 90.)
Regarding Claim 5: “The head mounted display system according to claim 1 wherein the computational device is operable to generate an image for each pupil of the user, each generated image corresponding to the virtual viewpoint of the respective pupil of the user and each generated image is displayed to the respective pupil of the user providing the user with a stereoscopic image.” (“the techniques described herein can be used to generate stereo images … to provide different output video for each of the two eyes.” Akeley, Paragraphs 110-111.)
Regarding Claim 6: “The head mounted display system according to claim 1 wherein the computational device is mounted to the display.” (“Augmented-reality glasses 202 may include various elements such as a computing system 208” Devries, Paragraph 26 and statement of motivation in Claim 1.)
Regarding Claim 7: “The head mounted display system according to claim 1, wherein the computational device is further operable to obtain an inter-pupil distance between the pupils of the user and determine the transformation further based on the inter-pupil distance.” (It is “apparent to those of skill in the art” that distance between the pupils is used as a representation of location of each of the eyes/pupils (see Specification Paragraphs 27-29). Prior art teaches this same application: “the processor may use the vantages to generate a stereo effect, by reproducing views at locations corresponding to the viewer's two eyes,” where eye locations account for the distance between the pupils of the user. Akeley, Paragraphs 19, 111-112.)
Regarding Claim 8: “The head mounted display system according to claim 1 wherein the computational device is connected to the display by a wire tether.” (See use of wired computing devices in Devries, Fig. 3 and networked computing devices in Fig. 7. “It will be appreciated that the network connections shown are exemplary and other means ( e.g., wired or wireless) of establishing a communications link between the computers 1002 and 1060 may be used when carrying out an aspect of an embodiment.” Devries, Paragraph 54. See statement of motivation in Claim 1.)
Regarding Claim 9: “The head mounted display system according to claim 1 wherein the computational device is wirelessly connected to the display.” (See use of wired computing devices in Devries, Fig. 3 and networked computing devices in Fig. 7. “It will be appreciated that the network connections shown are exemplary and other means ( e.g., wired or wireless) of establishing a communications link between the computers 1002 and 1060 may be used when carrying out an aspect of an embodiment.” Devries, Paragraph 54. See statement of motivation in Claim 1.)
Regarding Claim 10: “The head mounted display system of claim 1 wherein the selected virtual locations of the pupils of the user are selected by the user.” (“a viewer can freely position his or her head to view the environment from the corresponding position and viewing direction. … The viewing experience is immersive, meaning that the viewer sees the environment from his or her position and orientation as though he or she were actually in the scene at that position and orientation,” and thus a different viewing location of the pupils. Akely, Paragraph 90.)
Regarding Claim 11: “The head mounted display system of claim 1,
wherein the plurality of video cameras have fixed locations relative to the display, and (See cameras positioned at the same location as the display both for fixing the display location with respect to the object and with respect to the eye in Devries, Figs. 2-3 and statement of motivation in Claim 1.)
wherein the computational device is operable to determine the fixed respective fields of view of the plurality of video cameras relative to the pupils of the user based on an inter-pupil distance and an eye-to-display distance.” (“For playback to be immersive, the images presented to the viewer's eyes are ideally correct for both the position and orientation of his eyes. In general, the position and orientation of an eye will not match that of any camera, so it may be necessary to compute [reproject] the eye's image from one or more camera images at position(s) and/or orientation(s) that are different from those of the eye.” Akeley, Paragraphs 91-92, 112. “During reprojection, each pixel in a camera image 210 may be mapped to a corresponding location (typically not a pixel center) in the reprojected eye image 220,” for display “and then projecting that scene point to the eye image, as depicted in FIG. 2,” which accounts for the position of each eye with the exact inter-pupil distance, each image vantage, and each used display as the reprojection plane. Akeley, Paragraph 76. See a similar embodiment in Devries, Figs. 2-3 and statement of motivation in Claim 1.)
Claim 12, “A method of operating, by a processor, a head mounted display system,” is rejected for reasons stated for claim 1, because the method elements of Claim 12 are implemented by the system elements of Claim 1.
Claim 13 is rejected for reasons stated for Claim 5 in view of the Claim 12 rejection.
Claim 14 is rejected for reasons stated for Claim 7 in view of the Claim 12 rejection.
Claim 15 is rejected for reasons stated for Claims 1 and 4 in view of the Claim 12 rejection.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
US 20200018975 to Pombo (“Pombo”) relevant for teaching assembly of an HMD with relative locations of cameras and displays defined at the time of manufacturing.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MIKHAIL ITSKOVICH whose telephone number is (571)270-7940. The examiner can normally be reached Mon. - Thu. 9am - 8pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Joseph Ustaris can be reached at (571)272-7383. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MIKHAIL ITSKOVICH/Primary Examiner, Art Unit 2483
Prosecution Insights