DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/process/file/efs/guidance/eTD-info-I.jsp.
Claims 1–20 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1–21 of U.S. Patent No. 11,138,793 B2. Although the claims at issue are not identical, they are not patentably distinct from each other because their subject matter substantially overlaps.
Claims 1–20 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1–18 of U.S. Patent No. 12,182,945 B2. Although the claims at issue are not identical, they are not patentably distinct from each other because their subject matter substantially overlaps.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1–3, 7, and 8 are rejected under 35 U.S.C. 103(a) as being unpatentable over Border (US 2013/0278631 A1), Aksoy (US 2017/0148215 A1), and Schowengerdt (US 2015/0234254 A1).
Examiner notes Applicant’s problem being solved is extensively noted in the prior art literature. Applicant’s Specification makes clear eye strain and fatigue are problems created by virtual image content forcing the viewer to adjust simultaneous to disparate depth planes in 3D imaging applications (see published Spec., ¶ 0035). Equivalently, as just one example, “Hua” (Hua et al., “Augmented Reality: Easy on the Eyes,” Optics and Photonics News, February 2015) describes eye fatigue caused by accommodation-convergence cue discrepancy caused by the eyes attempting to focus on multiple depth planes simultaneously in augmented reality applications. Therefore, given the above evidence, Applicant’s recognition of the problem is addressed by the prior art. Turning now to Applicant’s claimed solution, i.e. moving objects onto a same depth plane so that the eyes do not have to focus simultaneously on multiple depth planes, Examiner notes the solution appears quite obvious given the known problem. By analogy, a man goes to the doctor and says, “It hurts when I do this.” The doctor replies, “Then don’t do that.” (comedically made famous by Smith and Dale, “Dr. Kronkheit and His Only Living Patient”, 1920s). Here, the problem is that it hurts the viewer’s eyes when virtual content and real-world content are presented in different depth planes. Applicant’s solution is to not do that. An important question is whether the prior art teaches how to present virtual content at the same depth the viewer is looking. The prior art demonstrates that such a solution is known. As explained in the rejections infra, Border, for example, teaches eye tracking to determining eye direction and vergence for obtaining a three-dimensional fixation point of a user of augmented reality eyewear. Furthermore, as the rejection, infra, explains, Border also teaches adjusting depth cues for the virtual object so that the virtual object is placed at the same depth as the user’s vergence. Therefore, given the above rationale, Applicant’s solution is addressed by the prior art.
Regarding claim 1, the combination of Border, Aksoy, and Schowengerdt teaches or suggests a display system comprising one or more processors and one or more computer storage media storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising: obtaining location information associated with a first virtual object to present to a user, wherein the first virtual object is configured to be presented, via a display device (Border, ¶¶ 1255 and 1262: teach determining eye direction and vergence for obtaining a three-dimensional coordinate for displaying a virtual object), with depth cues associated with a depth from the user (Border, ¶¶ 1255 and 1262: teach determining eye direction and vergence for obtaining a three-dimensional coordinate for displaying a virtual object; Examiner interprets this limitation as saying that when a user fixates on an object in an image, the claimed system presents a pop-up information bubble (or similar) related to the object at the same depth, but obviously laterally offset from the object so as not to occlude the object of interest; Examiner finds the technology described in Border, and other prior art of record for that matter, is capable of placing a virtual object at any position (x, y, z) in any image; Examiner further finds the described problem, placing a virtual object in the same depth plane as an object of interest determined by fixation, involves a simple solution of combining the prior art techniques of determining fixation x-, y-, z-coordinates and positioning virtual objects in a scene using x-, y-, z-coordinates), wherein the depth cues comprise accommodation cues and binocular cues, wherein the accommodation cues include wavefront divergence of light being output to the user, and the binocular cues include views of the virtual object that are being provided to each eye of the user (Examiner notes these are cues (characteristics) of the virtual object, as opposed to cues (characteristics) of the eyes; This can be confusing because, strictly speaking, the eyes and brain are typically the object of (interpreter of) the cues; Border, ¶ 1260: teaches adjusting the relative lateral positions of the binocular images of the virtual object, which is a binocular depth cue; Border, ¶ 1262: teaches using focus as a depth cue for a virtual object, i.e. accommodation cue; Aksoy, ¶ 0044: discloses a system that adjusts either accommodation or vergence cues (binocular) when there is an accommodation-vergence mismatch; Examiner notes Aksoy’s solution is a physical solution in which the optics block is moved to adjust the binocular cues to the eyes; Border, ¶ 1260: teaches adjusting the relative lateral positions of the binocular images, which is a binocular depth cue; Border, ¶ 1262: teaches using focus as a depth cue for a virtual object, i.e. accommodation; Therefore, Border teaches electrical cue adjustments, rather than physical cue adjustments; Examiner finds the skilled artisan would be capable of substituting Aksoy’s physical adjustments for Border’s graphical adjustments to achieve a solution to the well-known problem of accommodation-vergence mismatch; Schowengerdt, ¶ 0075: teaches a waveguide layer arrangement for see-through augmented reality glasses wherein the focal planes (depth planes) are defined by the waveguide layers; Schowengerdt, ¶ 0111: teaches adjusting the waveguide outputs to affect accommodation depth cues; Therefore, in combination, the focal depth plane can be adjusted as in Aksoy and Border using Schowengerdt’s wavefront divergence using waveguide layers; Schowengerdt, ¶ 0111: teaches adjusting the waveguide outputs to affect accommodation depth cues); adjusting binocular cues of the first virtual object to correspond with binocular cues of a fixation point, the fixation point indicating a three-dimensional location being fixated upon by the eyes of the user (Border, ¶ 1251: teaches eye tracking to determining eye direction and vergence for obtaining a three-dimensional fixation point of a user of augmented reality eyewear; Aksoy, ¶ 0021: teaches eye tracking for determining “gaze lines that identify a location or a position in a virtual scene presented by a headset where the user is looking.” Aksoy, ¶ 0022: explains the determination of vergence “is where the user is looking and is also typically the location where the user’s eyes are focused (i.e., accommodation of the eyes); Border, ¶¶ 1260–1261: teach adjusting depth cues for the virtual object so that the virtual object is placed at the same depth of focus as the user’s eyes; Border, ¶ 1260: teaches adjusting the relative lateral positions of the binocular images, which is a binocular depth cue; Border, ¶ 1262: teaches using focus as a depth cue for a virtual object, i.e. accommodation), wherein the accommodation cues of the first virtual object are retained, and wherein, after adjusting the binocular cues, the accommodation and vergence cues are mismatched (Aksoy, ¶ 0044: discloses a system that adjusts either accommodation or vergence cues (binocular) when there is an accommodation-vergence mismatch; Examiner makes an interpretative finding with respect to this limitation consistent with Banks et al. and Rushton et al. cited under the Conclusion Section of this Action; Specifically, because, as Banks explains, accommodation distance does not change while vergence distance varies depending on the depth of the object in the 3D scene, Examiner finds this limitation is describing that, instead of displaying the virtual object at the nominal depth plane (e.g. the display screen plane), the virtual object’s nominal depth plane is overridden in favor of a depth at which the viewer is viewing other 3D content; In order to avoid displaying the virtual object (such as a pop-up window describing the object) at a depth different from the actual object, which would be visually disturbing, the invention places the pop-up (or other virtual content) at the same depth as the object under consideration; Controlling depth in this way requires accommodation-vergence mismatch, thus producing an increased accommodation-vergence mismatch between the eyes’ accommodative distance and vergence distance; Border, ¶¶ 1260–1261: teach adjusting depth cues for the virtual object so that the virtual object is placed at the same depth of focus as the user’s eyes; Border, ¶ 1262: teaches using focus as a depth cue for a virtual object, i.e. accommodation); and causing presentation to the user, via a display device, of the first virtual object with the adjusted binocular cues (Border, ¶ 1251: teaches the vergence compensation scheme is for displaying an adjusted virtual image into a real image or scene).
One of ordinary skill in the art, before the effective filing date of the claimed invention, would have been motivated to combine the elements taught by Border, with those of Aksoy, because both references are drawn to the same field of endeavor, and because determining a three-dimensional fixation point of a viewer’s eyes (eye tracking) is important in augmented reality viewing systems that desire to place helpful augmented content at the correct location in the three-dimensional scene under view (e.g., Border, ¶¶ 1260–1261). This rationale applies to all combinations of Border and Aksoy used in this Office Action unless otherwise noted.
One of ordinary skill in the art, before the effective filing date of the claimed invention, would have been motivated to combine the elements taught by the combination of Border and Aksoy, with those of Schowengerdt, because the references are drawn to the same field of endeavor, and because utilizing stacked waveguides to create various depth planes allows for better accommodation control (Schowengerdt, ¶ 0111). This rationale applies to all combinations of Border, Aksoy, and Schowengerdt used in this Office Action unless otherwise noted.
Regarding claim 2, the combination of Border, Aksoy, and Schowengerdt teaches or suggests the display system of claim 1, wherein the depth cues further comprise size cues (Official Notice: Applicant did not invent depicting an object as bigger to cue to the eye that it is closer).
Regarding claim 3, the combination of Border, Aksoy, and Schowengerdt teaches or suggests the display system of claim 1, wherein the operations further comprise: determining a first perceived depth associated with the wavefront divergence of light being output to the user; determining a second perceived depth associated with binocular disparity; and determining an accommodation-vergence mismatch by calculating a mismatch between the first and second perceived depths (Examiner interprets the first depth as the virtual object depth and the second depth as the gaze depth (real-world or other virtual object being fixated on); Border, ¶ 1251: teaches eye tracking to determining eye direction and vergence for obtaining a three-dimensional fixation point of a user of augmented reality eyewear; Aksoy, ¶ 0021: teaches eye tracking for determining “gaze lines that identify a location or a position in a virtual scene presented by a headset where the user is looking.” Aksoy, ¶ 0022: explains the determination of vergence “is where the user is looking and is also typically the location where the user’s eyes are focused (i.e., accommodation of the eyes); Border, ¶¶ 1260–1261: teach adjusting depth cues for the virtual object so that the virtual object is placed at the same depth of focus as the user’s eyes; Border, ¶ 1260: teaches adjusting the relative lateral positions of the binocular images, which is a binocular depth cue; Border, ¶ 1262: teaches using focus as a depth cue for a virtual object, i.e. accommodation).
Regarding claim 7, the combination of Border, Aksoy, and Schowengerdt teaches or suggests the display system of claim 1, further comprising the display device, the display device comprising: a plurality of stacked waveguides forming a display area and providing a view of an ambient environment through the display area, further comprising light outcoupling optical elements in the display area, the light outcoupling optical elements configured to output light having image information (Schowengerdt, ¶ 0075: teaches a waveguide layer arrangement for see-through augmented reality glasses wherein the focal planes (depth planes) are defined by the waveguide layers; Schowengerdt, ¶ 0111: teaches adjusting the waveguide outputs to affect accommodation depth cues).
Regarding claim 8, the combination of Border, Aksoy, and Schowengerdt teaches or suggests the display system of claim 7, wherein at least some waveguides of the plurality of stacked waveguides are configured to output light with different wavefront divergence than other waveguides, wherein different amounts of wavefront divergence are associated with different depth planes (Schowengerdt, ¶ 0075: teaches a waveguide layer arrangement for see-through augmented reality glasses wherein the focal planes (depth planes) are defined by the waveguide layers; Schowengerdt, ¶ 0111: teaches adjusting the waveguide outputs to affect accommodation depth cues).
Claims 4, 5, 9, 13, 17, 18, and 20 are rejected under 35 U.S.C. 103(a) as being unpatentable over Border, Aksoy, Schowengerdt, and Rushton et al., “Developing visual systems and exposure to virtual reality and stereo displays: some concerns and speculations about the demands on accommodation and vergence,” Applied Ergonomics 30 (1999).
Regarding claim 4, the combination of Border, Aksoy, Schowengerdt, and Rushton teaches or suggests the display system of claim 3, wherein the operations further comprise: determining the accommodation-vergence mismatch is greater than a threshold; and in response to determining the accommodation-vergence mismatch is greater than the threshold, further adjusting the binocular cues such that the mismatch is less than or equal to the threshold (Aksoy, ¶ 0044: discloses a system that adjusts either accommodation or vergence cues (binocular) when there is an accommodation-vergence mismatch; Examiner notes Aksoy’s solution is a physical solution in which the optics block is moved to adjust the binocular cues to the eyes; Border, ¶ 1260: teaches adjusting the relative lateral positions of the binocular images, which is a binocular depth cue; Border, ¶ 1262: teaches using focus as a depth cue for a virtual object, i.e. accommodation; Therefore, Border teaches electrical cue adjustments, rather than physical cue adjustments; Examiner finds the skilled artisan would be capable of substituting Aksoy’s physical adjustments for Border’s graphical adjustments to achieve a solution to the well-known problem of accommodation-vergence mismatch; While Border and Aksoy suggest adjusting vergence and accommodation cues in general to help with eye strain, Rushton explicitly explains that because the accommodation plane is fixed for virtual content, it has been suggested to constrain the binocular disparity of virtual objects to limit the accommodation-vergence mismatch to a level necessary to assure the comfort and safety of users).
One of ordinary skill in the art, before the effective filing date of the claimed invention, would have been motivated to combine the elements taught by the combination of Border, Aksoy, and Schowengerdt, with those of Rushton, because the references are drawn to the same field of endeavor, and because improving viewer comfort and safety in the use of VR displays can be achieved by constraining the disparity of virtual objects (Rushton, Section 5.4, citing Peli’s work). This rationale applies to all combinations of Border, Aksoy, Schowengerdt, and Rushton used in this Office Action unless otherwise noted.
Regarding claim 5, the combination of Border, Aksoy, Schowengerdt, and Rushton teaches or suggests the display system of claim 4, wherein the binocular cues are adjusted such that the second perceived depth matches the first perceived depth (Border, ¶ 1260: teaches adjusting the relative lateral positions of the binocular images, which is a binocular depth cue; Border, ¶¶ 1260–1261: teach adjusting depth cues for the virtual object so that the virtual object is placed at the same depth of focus as the user’s eyes).
Regarding claim 9, the combination of Border, Aksoy, Schowengerdt, and Rushton teaches or suggests a system comprising: a display device comprising a stack of waveguides, the stack of waveguides forming a display area and providing a view of an ambient environment through the display area, wherein a first waveguide of the stack of waveguides is configured to output light with a first wavefront divergence and wherein a second waveguide of the stack of waveguides is configured to output light with a second wavefront divergence (Schowengerdt, ¶ 0075: teaches a waveguide layer arrangement for see-through augmented reality glasses wherein the focal planes (depth planes) are defined by the waveguide layers; Schowengerdt, ¶ 0111: teaches adjusting the waveguide outputs to affect accommodation depth cues); one or more processors configured to provide access to information from a map of the ambient environment, the map comprising respective three-dimensional locations for a first virtual object and a second virtual object, wherein a location for the first virtual object is associated with the first wavefront divergence and has associated binocular depth cues, and wherein the second virtual object is associated with the second wavefront divergence and second binocular depth cues (Border, ¶¶ 1255 and 1262: teaches determining eye direction and vergence for obtaining a three-dimensional coordinate for displaying a virtual object); and one or more computer storage media storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising: determining an indication of a fixation point of a user's eyes, wherein the fixation point corresponds to a location of the first virtual object; obtaining, via the map, location information associated with the second virtual object, wherein the second virtual object is configured to be presented, via the display device, with the second wavefront divergence from the second waveguide and with the second binocular depth cues (Border, ¶ 1251: teaches eye tracking to determining eye direction and vergence for obtaining a three-dimensional fixation point of a user of augmented reality eyewear; Aksoy, ¶ 0021: teaches eye tracking for determining “gaze lines that identify a location or a position in a virtual scene presented by a headset where the user is looking.” Aksoy, ¶ 0022: explains the determination of vergence “is where the user is looking and is also typically the location where the user’s eyes are focused (i.e., accommodation of the eyes); Border, ¶ 1260: teaches adjusting the relative lateral positions of the binocular images of the virtual object, which is a binocular depth cue; Border, ¶ 1262: teaches using focus as a depth cue for a virtual object, i.e. accommodation cue); subsequently changing the second wavefront divergence associated with the second virtual object to correspond with the first wavefront divergence of the first virtual object, wherein the second binocular depth cues of the second virtual object are retained (Border, ¶¶ 1260–1261: teach adjusting depth cues for the virtual object so that the virtual object is placed at the same depth of focus as the user’s eyes; Border, ¶ 1260: teaches adjusting the relative lateral positions of the binocular images, which is a binocular depth cue; Aksoy, ¶ 0044: discloses a system that adjusts either accommodation or vergence cues (binocular) when there is an accommodation-vergence mismatch; Examiner notes Aksoy’s solution is a physical solution in which the optics block is moved to adjust the binocular cues to the eyes; Border, ¶ 1260: teaches adjusting the relative lateral positions of the binocular images, which is a binocular depth cue; Border, ¶ 1262: teaches using focus as a depth cue for a virtual object, i.e. accommodation; Therefore, Border teaches electrical cue adjustments, rather than physical cue adjustments; Examiner finds the skilled artisan would be capable of substituting Aksoy’s physical adjustments for Border’s graphical adjustments to achieve a solution to the well-known problem of accommodation-vergence mismatch; While Border and Aksoy suggest adjusting vergence and accommodation cues in general to help with eye strain, Rushton explicitly explains that because the accommodation plane is fixed for virtual content, it has been suggested to constrain the binocular disparity of virtual objects to limit the accommodation-vergence mismatch to a level necessary to assure the comfort and safety of users); and causing presentation to the user, via the first waveguide of the display device, of the second virtual object with the changed second wavefront divergence (Border, ¶ 1251: teaches the vergence compensation scheme is for displaying an adjusted virtual image into a real image or scene).
Regarding claim 13, the combination of Border, Aksoy, Schowengerdt, and Rushton teaches or suggests the system of claim 9, wherein the operations further comprise: determining that subsequently changing the second wavefront divergence would increase an accommodation-vergence mismatch beyond a threshold; and adjusting the second binocular depth cues of the second virtual object to reduce the accommodation-vergence mismatch to below the threshold (Aksoy, ¶ 0044: discloses a system that adjusts either accommodation or vergence cues (binocular) when there is an accommodation-vergence mismatch; Examiner notes Aksoy’s solution is a physical solution in which the optics block is moved to adjust the binocular cues to the eyes; Border, ¶ 1260: teaches adjusting the relative lateral positions of the binocular images, which is a binocular depth cue; Border, ¶ 1262: teaches using focus as a depth cue for a virtual object, i.e. accommodation cue; Therefore, Border teaches electrical cue adjustments, rather than physical cue adjustments; Examiner finds the skilled artisan would be capable of substituting Aksoy’s physical adjustments for Border’s graphical adjustments to achieve a solution to the well-known problem of accommodation-vergence mismatch; While Border and Aksoy suggest adjusting vergence and accommodation cues in general to help with eye strain, Rushton explicitly explains that because the accommodation plane is fixed for virtual content, it has been suggested to constrain the binocular disparity of virtual objects to limit the accommodation-vergence mismatch to a level necessary to assure the comfort and safety of users).
Regarding claim 17, the combination of Border, Aksoy, Schowengerdt, and Rushton teaches or suggests the system of claim 9, wherein the second binocular depth cues further comprise size cues (Official Notice: Applicant did not invent depicting an object as bigger to cue to the eye that it is closer).
Regarding claim 18, the combination of Border, Aksoy, Schowengerdt, and Rushton teaches or suggests a display system comprising a display device configured to present virtual objects to a user on a plurality of depth planes (Schowengerdt, ¶ 0075: teaches a waveguide layer arrangement for see-through augmented reality glasses wherein the focal planes (depth planes) are defined by the waveguide layers; Schowengerdt, ¶ 0111: teaches adjusting the waveguide outputs to affect accommodation depth cues); one or more processors; and one or more computer storage media storing instructions that when executed by the display system, cause the display system to perform operations comprising: accessing a map of the world that comprises respective locations for a first virtual object and a second virtual object, the first virtual object being presented on a first depth plane (Border, ¶¶ 1255 and 1262: teach determining eye direction and vergence for obtaining a three-dimensional coordinate for displaying a virtual object; Border, ¶ 1251: teaches eye tracking to determining eye direction and vergence for obtaining a three-dimensional fixation point of a user of augmented reality eyewear; Aksoy, ¶ 0021: teaches eye tracking for determining “gaze lines that identify a location or a position in a virtual scene presented by a headset where the user is looking.” Aksoy, ¶ 0022: explains the determination of vergence “is where the user is looking and is also typically the location where the user’s eyes are focused (i.e., accommodation of the eyes); Border, ¶ 1260: teaches adjusting the relative lateral positions of the binocular images of the virtual object, which is a binocular depth cue; Border, ¶ 1262: teaches using focus as a depth cue for a virtual object, i.e. accommodation cue); determining an indication of a fixation point of the eyes of the user, the fixation point corresponding to the location of the first virtual object; and based on the determined indication of the fixation point, changing display properties of the second virtual object such that the second virtual object is displayed on the first depth plane, wherein the second virtual object was configured to be presented on a second depth plane of the plurality of depth planes based on the map, and wherein display of the second virtual object is changed to the first depth plane (Border, ¶¶ 1260–1261: teach adjusting depth cues for the virtual object so that the virtual object is placed at the same depth of focus as the user’s eyes; Border, ¶ 1260: teaches adjusting the relative lateral positions of the binocular images, which is a binocular cue; Aksoy, ¶ 0044: discloses a system that adjusts either accommodation or vergence cues (binocular) when there is an accommodation-vergence mismatch; Examiner notes Aksoy’s solution is a physical solution in which the optics block is moved to adjust the binocular cues to the eyes; Border, ¶ 1260: teaches adjusting the relative lateral positions of the binocular images, which is a binocular depth cue; Border, ¶ 1262: teaches using focus as a depth cue for a virtual object, i.e. accommodation cue; Therefore, Border teaches electrical cue adjustments, rather than physical cue adjustments; Examiner finds the skilled artisan would be capable of substituting Aksoy’s physical adjustments for Border’s graphical adjustments to achieve a solution to the well-known problem of accommodation-vergence mismatch; While Border and Aksoy suggest adjusting vergence and accommodation cues in general to help with eye strain, Rushton explicitly explains that because the accommodation plane is fixed for virtual content, it has been suggested to constrain the binocular disparity of virtual objects to limit the accommodation-vergence mismatch to a level necessary to assure the comfort and safety of users).
Regarding claim 20, the combination of Border, Aksoy, Schowengerdt, and Rushton teaches or suggests the display system of claim 18, wherein changing display properties comprises: determining that subsequently changing the display properties would increase an accommodation-vergence mismatch beyond a threshold; and adjusting one or more binocular cues of the second virtual object to reduce the accommodation-vergence mismatch to below the threshold (Aksoy, ¶ 0044: discloses a system that adjusts either accommodation or vergence cues (binocular) when there is an accommodation-vergence mismatch; Examiner notes Aksoy’s solution is a physical solution in which the optics block is moved to adjust the binocular cues to the eyes; Border, ¶ 1260: teaches adjusting the relative lateral positions of the binocular images, which is a binocular depth cue; Border, ¶ 1262: teaches using focus as a depth cue for a virtual object, i.e. accommodation; Therefore, Border teaches electrical cue adjustments, rather than physical cue adjustments; Examiner finds the skilled artisan would be capable of substituting Aksoy’s physical adjustments for Border’s graphical adjustments to achieve a solution to the well-known problem of accommodation-vergence mismatch; While Border and Aksoy suggest adjusting vergence and accommodation cues in general to help with eye strain, Rushton explicitly explains that because the accommodation plane is fixed for virtual content, it has been suggested to constrain the binocular disparity of virtual objects to limit the accommodation-vergence mismatch to a level necessary to assure the comfort and safety of users).
Claim 6 is rejected under 35 U.S.C. 103(a) as being unpatentable over Border, Aksoy, Schowengerdt, and Boelter (US 2014/0092006 A1).
Regarding claim 6, the combination of Border, Aksoy, Schowengerdt, and Boelter teaches or suggests the display system of claim 1, wherein the operations further comprise: determining that the user is alternating fixation, at greater than a threshold frequency (Boelter, ¶ 0026: teaches comparing the frequency of a viewer’s changing focus area to a threshold to filter out saccades from fixations), between the first virtual object and a second object, wherein the fixation point coincides with a three-dimensional location of the second object (Border, ¶ 1251: teaches eye tracking to determining eye direction and vergence for obtaining a three-dimensional fixation point of a user of augmented reality eyewear; Aksoy, ¶ 0021: teaches eye tracking for determining “gaze lines that identify a location or a position in a virtual scene presented by a headset where the user is looking.” Aksoy, ¶ 0022: explains the determination of vergence “is where the user is looking and is also typically the location where the user’s eyes are focused (i.e., accommodation of the eyes)), and wherein the second object is a virtual object or a real-world object (Border, ¶ 1251: teaches the vergence compensation scheme is for displaying an adjusted virtual image into a real image or scene), wherein adjusting the binocular cues of the first virtual object is in response to determining that the user is alternating fixation (Boelter, ¶ 0026: teaches comparing the frequency of a viewer’s changing focus area to a threshold to filter out saccades from fixations).
One of ordinary skill in the art, before the effective filing date of the claimed invention, would have been motivated to combine the elements taught by the combination of Border, Aksoy, and Schowengerdt, with those of Boelter, because the references are drawn to the same field of endeavor, and because differentiating between fixations and saccades is a key component to determining the gaze of a AR user (Boelter, ¶ 0026). This rationale applies to all combinations of Border, Aksoy, Schowengerdt, and Boelter used in this Office Action unless otherwise noted.
Claims 10, 12, and 19 are rejected under 35 U.S.C. 103(a) as being unpatentable over Border, Aksoy, Schowengerdt, Rushton, and Boelter (US 2014/0092006 A1).
Regarding claim 10, the combination of Border, Aksoy, Schowengerdt, Rushton, and Boelter teaches or suggests the system of claim 9, wherein determining the indication of the fixation point comprises determining that the user's eyes are fixating on the location for greater than a threshold duration (Boelter, ¶ 0026: teaches comparing the frequency of a viewer’s changing focus area to a threshold to filter out saccades from fixations; Boelter, ¶ 0017: teaches the user’s fixation can define an area of interest; Border, ¶ 0672: teaches gaze detection at a real-world object; Border, ¶ 1275: teaches a user gazing at a real-world object and the system providing virtual content to supplement the real world object).
One of ordinary skill in the art, before the effective filing date of the claimed invention, would have been motivated to combine the elements taught by the combination of Border, Aksoy, Schowengerdt, and Rushton, with those of Boelter, because the references are drawn to the same field of endeavor, and because differentiating between fixations and saccades is a key component to determining the gaze of a AR user (Boelter, ¶ 0026). This rationale applies to all combinations of Border, Aksoy, Schowengerdt, Rushton, and Boelter used in this Office Action unless otherwise noted.
Regarding claim 12, the combination of Border, Aksoy, Schowengerdt, Rushton, and Boelter teaches or suggests the system of claim 9, wherein the operations further comprise: identifying that the second wavefront divergence for the second virtual object is to be adjusted based on an indication of user preference or a determination that the second virtual object is being fixated on less than the first virtual object (Border, ¶¶ 1260–1261: teach adjusting depth cues for the virtual object so that the virtual object is placed at the same depth of focus as the user’s eyes; Border, ¶ 1260: teaches adjusting the relative lateral positions of the binocular images, which is a binocular depth cue; Border, ¶ 1262: teaches using focus as a depth cue for a virtual object, i.e. accommodation; Boelter, ¶ 0026: teaches comparing the frequency of a viewer’s changing focus area to a threshold to filter out saccades from fixations).
Regarding claim 19, the combination of Border, Aksoy, Schowengerdt, Rushton, and Boelter teaches or suggests the display system of claim 18, wherein determining the indication of the fixation point comprises determining that the fixation point falls on the first depth plane for greater than a threshold metric before changing display properties of the second virtual object (Boelter, ¶ 0026: teaches comparing the frequency of a viewer’s changing focus area to a threshold to filter out saccades from fixations).
Claim 11 is rejected under 35 U.S.C. 103(a) as being unpatentable over Border, Aksoy, Schowengerdt, Rushton, Boelter, and Starner (US 2015/0097772 A1).
Regarding claim 11, the combination of Border, Aksoy, Schowengerdt, Rushton, Boelter, and Starner teaches or suggests the system of claim 9, wherein subsequently changing the second wavefront divergence comprises determining that the user is alternating fixation, at greater than a threshold frequency, between the first virtual object and the second virtual object (Boelter, ¶ 0026: teaches comparing the frequency of a viewer’s changing focus area to a threshold; Examiner notes the frequency of alternating fixation can be due to an accommodation-vergence mismatch; Aksoy, ¶ 0044: discloses a system that adjusts either accommodation or vergence cues (binocular) when there is an accommodation-vergence mismatch; Examiner notes Aksoy’s solution is a physical solution in which the optics block is moved to adjust the binocular cues to the eyes; Border, ¶ 1260: teaches adjusting the relative lateral positions of the binocular images, which is a binocular depth cue; Border, ¶ 1262: teaches using focus as a depth cue for a virtual object, i.e. accommodation; Therefore, Border teaches electrical cue adjustments, rather than physical cue adjustments; Examiner finds the skilled artisan would be capable of substituting Aksoy’s physical adjustments for Border’s graphical adjustments to achieve a solution to the well-known problem of accommodation-vergence mismatch; While Border and Aksoy suggest adjusting vergence and accommodation cues in general to help with eye strain, Starner, ¶ 0114: teaches determining a maximum frequency of occurrence of eye movements between fixations).
One of ordinary skill in the art, before the effective filing date of the claimed invention, would have been motivated to combine the elements taught by the combination of Border, Aksoy, Schowengerdt, Rushton, and Boelter, with those of Starner, because the references are drawn to the same field of endeavor, because setting values of eye movement parameters in eye tracking systems (Starner, ¶ 0114) can help with inaccurate or unreliable noisy gaze signals (Starner, ¶ 0022), and because the combination is a mere combination of prior art elements, according to known methods, to yield a predictable result. This rationale applies to all combinations of Border, Aksoy, Schowengerdt, Rushton, Boelter, and Starner used in this Office Action unless otherwise noted.
Claim 14 is rejected under 35 U.S.C. 103 as being unpatentable over Border, Aksoy, Schowengerdt, Rushton, and Lapstun (US 2016/0139402 A1).
Regarding claim 14, the combination of Border, Aksoy, Schowengerdt, Rushton, and Lapstun teaches or suggests the system of claim 13, wherein the threshold of the accommodation-vergence is 0.33 diopter (Examiner notes this claim uses a term historically used to describe lens power or magnification; Diopter is the reciprocal of focal length; Lapstun, ¶ 0447: teaches ideal accommodation ranges related to normal focal length distances are 2-3 meters, which is 0.5 to 0.33 diopter and wherein farther distances do not allow for proper accommodation; Lapstun teaches various light field display technologies for creating wavefronts that represent depths of focus (e.g. ¶ 0011 and 0424). Lapstun also discusses vergence-accommodation conflicts using correctly centered spherical wavefronts (e.g. ¶ 0451); see also Rushton’s teachings explaining that because the accommodation plane is fixed for virtual content, it has been suggested to constrain the binocular disparity of virtual objects to limit the accommodation-vergence mismatch to a level necessary to assure the comfort and safety of users).
One of ordinary skill in the art, before the effective filing date of the claimed invention, would have been motivated to combine the elements taught by the combination of Border, Aksoy, Schowengerdt, and Rushton, with those of Lapstun, because the references are drawn to the same field of endeavor, because considerations of accommodation for head-worn displays such as wavefront displays is important (e.g. Lapstun, ¶ 0447), because the skilled artisan could use Lapstun’s diopter value for binocular distance to plug in for the constraint described by Rushton, and because the combination is a mere combination of prior art elements, according to known methods, to yield a predictable result. This rationale applies to all combinations of Border, Aksoy, Schowengerdt, Rushton, and Lapstun used in this Office Action unless otherwise noted.
Claims 15 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Border, Aksoy, Schowengerdt, Rushton, and Popovich (US 2018/0232048 A1).
Regarding claim 15, the combination of Border, Aksoy, Schowengerdt, Rushton, and Popovich teaches or suggests the system of claim 9, wherein the display device is a vari-focal display device (Popovich, ¶ 0037: teaches a waveguide display that provides multiple focal surfaces).
One of ordinary skill in the art, before the effective filing date of the claimed invention, would have been motivated to combine the elements taught by the combination of Border, Aksoy, Schowengerdt, and Rushton, with those of Popovich, because the references are drawn to the same field of endeavor, because stacked waveguides are a well-known approach to achieving better outcomes when addressing accommodation-vergence conflicts (e.g. Popovich, ¶ 0037), and because the combination is a mere combination of prior art elements, according to known methods, to yield a predictable result. This rationale applies to all combinations of Border, Aksoy, Schowengerdt, Rushton, and Popovich used in this Office Action unless otherwise noted.
Regarding claim 16, the combination of Border, Aksoy, Schowengerdt, Rushton, and Popovich teaches or suggests the system of claim 9, wherein at least some waveguides of the stack of waveguides are configured to output light of different component colors than other waveguides (Popovich, ¶ 0235: teaches the multiple stacked waveguides can correspond to different color components).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Lapstun (US 2016/0139402 A1) teaches various light field display technologies for creating wavefronts that represent depths of focus (e.g. ¶ 0011 and 0424). The reference also discusses vergence-accommodation conflicts using correctly centered spherical wavefronts (e.g. ¶ 0451).
Lapstun (US 2017/0214907 A1) teaches substantially similar subject matter to that described above.
Schowengerdt (US 2015/0205126 A1) teaches subject matter closely related to the instant invention (e.g. Abstract).
Raffle (US 2016/0077337 A1) teaches thresholds of frequency of occurrence of eye movements between fixations (¶ 0121).
Chen (US 2017/0116459 A1) teaches eye tracking and vergence concepts (¶ 0037).
Horesh (US 2016/0134863 A1) teaches eye tracking calibration and saccade movement between two fixations (e.g. ¶ 0016).
Merle (US 2015/0123820 A1) teaches calculating fixation/saccades ratio for a predetermined time in an eye tracking algorithm (¶ 0029).
Hua et al., “Augmented Reality: Easy on the Eyes,” Optics and Photonics News, February 2015.
Banks, et al., “Insight into Vergence-Accommodation Mismatch,” Proc. SPIE, May 16, 2013. This reference provides a definition of vergence-accommodation conflict or mismatch by explaining, “Stereo viewing creates inconsistencies between vergence and accommodative distances because the vergence distance varies depending on the image contents while the accommodative distance remains constant.” (Caption for Fig. 1).
Rushton et al., “Developing visual systems and exposure to virtual reality and stereo displays: some concerns and speculations about the demands on accommodation and vergence,” Applied Ergonomics 30 (1999). Discusses how the natural environment part of augmented reality reinforces accommodation-vergence similarity while the virtual reality imagery requires accommodation-vergence mismatch. “Given an object on a stereo display, accommodation and vergence responses will be mismatched and the visual system may only reduce this mismatch through selecting a compromise state.” See also Section 5.4, explaining the benefits of constraining disparity of objects in a VR display “by limiting the vergence-accommodation mismatch.”
Hoffman et al., “Vergence-accommodation conflicts hinder visual performance and cause visual fatigue,” June 1, 2010.
“Accommodation and vergence in 3D VR,” Reddit blog, Aug. 30, 2014.
Hunter, “Vergence-accommodation conflict is a bitch -– here’s how to design around it,” Nov. 8, 2015.
Kramida et al., “Resolving the Vergence-Accommodation Conflict in Head Mounted Displays,” Aug. 27, 2015.
M. Vinnikov and R. S. Allison, “Gaze-contingent depth of field in realistic scenes,” in Proceedings of the Symposium on Eye Tracking Research and Applications - ETRA ’14. New York, New York, USA: ACM Press, Mar. 2014, pp. 119–126. [Online]. Available: http://dl.acm.org/citation.cfm?id=2578153.2578170. The publication’s Abstract states, in pertinent part, “It is believed that, by artificially simulating focal blur and natural accommodation, it is possible to compensate for the vergence-accommodation conflict and alleviate these symptoms. We hypothesized that it is possible to compensate for conflict with a fixed accommodation cue by adding simulated focal blur according to instantaneous fixation.”
Lanier (US 2017/0154464 A1) teaches thresholded accommodation cues (¶ 0036).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Michael J Hess whose telephone number is (571)270-7933. The examiner can normally be reached Mon - Fri 9:00am-5:30pm.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, William Vaughn can be reached on (571)272-3922. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8933.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHAEL J HESS/Examiner, Art Unit 2481
Prosecution Insights