DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-2, 4, 6-15, 17-18 and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Juenger (US 2018/0035102), and further in view of Vlaskamp (US 2021/0181840).
Regarding claim 1, Juenger teaches a head-wearable device (head-mounted display device 118, fig. 1 and [0021]), configured to be worn by a user ([0021]: the housing 112 is configured to be worn on the head of a viewer 108 (e.g., as a head-mounted display device 118), such as through configuration as goggles, glasses, contact lens, and so forth), for presenting an artificial- reality environment ([0019]: environment 100 includes a computing device 102 configured for use in augmented reality and/or virtual reality scenarios; [0030]: FIG. 2 illustrates an example 200 of digital content that can be rendered for viewing in an augmented reality or virtual reality environment), the head-wearable device comprising:
one or more image-projection systems (display device 116 can be configured to use projectors to render digital content 106; [0024]: The display device 116 is configurable in a variety of ways to support rendering of the digital content 106. Examples of which include … a light field display for use on a head-mounted display in which a viewer may see through portions of the display (e.g., as part of an augmented reality scenario), projectors);
one or more programs ([0059] and [0061]), wherein the one or more programs are stored in memory (computer-readable media 706, fig. 7 and [0061]) and configured to be executed by one or more processors (processing system 704, fig. 7 and [0056]), the one or more programs including instructions for:
while an image is being presented to a user's first eye using a first image- projection system of the head-wearable device ([0015]: a computing device (e.g., a head-mounted display device) causes display of stereoscopic digital content, which includes one or more left images rendered for a left eye of a viewer; [0025]: display device 116 is implemented as a stereoscopic display 103 that can provide three-dimensional (3D) content to viewers, such as images (e.g., stereoscopic imagery) and/or video effective to cause a viewer to be able to perceive depth within the content when displayed. The stereoscopic display 103 may be implemented in a variety of different ways, such as a liquid crystal display on a silicon panel or a micro O-LED. Generally, the stereoscopic display 103 enables display of different stereoscopic imagery to each eye of the viewer 108. For example, digital content may be rendered for the viewer that includes a left image rendered for the viewer's left eye and a right image rendered for the viewer's right eye; [0050]: At 602, display stereoscopic digital content that includes one or more left images and one or more right images is displayed. For example, display device 116 of computing device 102 causes display of digital content 106, which includes one or more left images for a left eye of a viewer 108, and one or more right images for a right eye of the viewer 108) and the image is being presented to a user's second eye using a second image-projection system of the head-wearable device ([0015]: a computing device (e.g., a head-mounted display device) causes display of stereoscopic digital content, which includes one or more right images rendered for a right eye of the viewer; [0050]: At 602, display stereoscopic digital content that includes one or more left images and one or more right images is displayed. For example, display device 116 of computing device 102 causes display of digital content 106, which includes one or more left images for a left eye of a viewer 108, and one or more right images for a right eye of the viewer 108):
selecting one or both of (i) a selected point in time (at periodic time intervals such as when the computing device 102 is powered on, and every 30 seconds thereafter) at which to present a realignment pattern (alignment pattern) via the head-wearable device ([0033]: alignment module 126 causes alignment patterns 128 to be generated within the digital content 106. In some cases, the alignment module 126 may generate the alignment patterns 128 at periodic time intervals, such as when the computing device 102 is powered on, and every 30 seconds thereafter)
presenting, via the head-wearable device, the realignment pattern at one or both of the selected point in time (periodic time intervals) ([0029]: the alignment module 126 is configured to generate alignment patterns 128 within the digital content 106 such that the alignment patterns 128 are within the field of view of the viewer 108; [0037]: an alignment pattern 402 is generated or embedded within the calendar tile 202 of digital content 106, and includes an “egg crate” pattern. It is to be noted, however, that alignment pattern 128 may be generated in a variety of different ways, using different types of markings, lines, shapes, and so forth); and
modifying presentation characteristics for the first image-projection system or the second image-projection system based on the presenting of the realignment pattern ([0018]: the digital content is realigned by adjusting at least one of the left and right images of the stereoscopic digital content based on the detected alignment patterns; [0028]: the alignment module adjusts at least one of the left or right images of the digital content 106 in order to realign the digital content 106. Alternately, to realign the digital content 106, the alignment module 126 can guide a physical realignment of display device 116; [0034]: After the alignment patterns 128 are generated, the cameras 120 are controlled to detect the alignment patterns 128 in each of the left and right images of the digital content 106. The location and position of the alignment patterns 128 may then be used to automatically realign the left and right images of the digital content 106; [0046]: Alignment module 126 knows the position of the alignment patterns 128 within the digital content 106, and thus alignment module 126 can use the detected alignment patterns to calculate an amount of offset of the alignment pattern 128 in both of the left and right images of the digital content 106. Then, alignment module 126 uses the offset to adjust at least one of the left or right images of the digital content to perform the realignment; [0053]: At 608, the stereoscopic digital content is realigned by adjusting at least one of the left and right images of the stereoscopic digital content based on the detected alignment patterns. For example, alignment module 126 adjusts at least one of the left and right images of the digital content 106 in order to align the images).
Juenger does not explicitly teach selecting a location within the image at which the realignment pattern should be presented; and presenting, via the head-wearable device, the realignment pattern at the selected location.
Vlaskamp teaches selecting a location within the image at which the realignment pattern should be presented ([0080]: select a first vertical position at which to present the left-eye alignment marker; [0081]: select a second vertical position at which to present the right-eye alignment marker; [0087]: pseudo- or quasi-randomly select the second vertical position at which to present the right-eye alignment marker; [0304]: the vertical positions of the alignment markers 1502 and 1504 may be randomized, pseudorandomized, or quasi-randomized. For example, the system may randomly, pseudorandomly, or quasi-randomly select the vertical positions of the alignment markers 1502 and 1504 from a predetermined range of vertical positions. In some embodiments, the system may randomly, pseudorandomly, or quasi-randomly select the offset between the alignment markers 1502 and 1504 from a predetermined range of offsets. In some embodiments, the display system may be configured to randomly, pseudorandomly, or quasi-randomly select the offset based on pseudorandom number generators, such as a Mersenne Twister, Xorshift, and so on. In some embodiments, the pseudorandom number generators may generate the offset values directly. In some other embodiments, each offset may be associated with a number, or set of numbers, and the generation of a number by a pseudorandom number generators may be used to select the offset associated with that generated number or with the set to which that generated number belongs (e.g., each offset may be associated with one or more unique numbers or sets of numbers, a number may be generated using the pseudorandom generator, and the offset to be applied may be selected based on the correspondence between that generated number (or the set to which the number belongs) and the numbers (or sets of numbers) associated with the offsets); [0311]: the HMD may display one or both of alignment markers 1502 and 1504 at randomly-, pseudorandomly-, or quasi-randomly-selected vertical positions); and presenting, via the head-wearable device, the realignment pattern at the selected location ([0082]: wherein to provide, with the left-eye display and the right-eye display, the left-eye alignment marker and the right-eye alignment marker, respectively, the at least one processor is configured to; [0083]: provide, with the left-eye display, the left-eye alignment marker at the first vertical position; [0084]: provide, with the right-eye display, the right-eye alignment marker at the second vertical position; [0085]: wherein to select the first vertical position at which to present the left-eye alignment marker and select the second vertical position at which to present the right-eye alignment marker, the at least one processor is configured to; [0086]: pseudo- or quasi-randomly select the first vertical position at which to present the left-eye alignment marker; [0303]: When a user wearing the HMD views the screens 1600a and 1600b simultaneously (e.g., with their left and right eyes respectively), the vertical alignment markers 1506a and 1506b may appear to the user as being fused together (e.g., as mark 1506). However, at least because the left-eye and right-eye horizontal alignment markers 1502 and 1504 are not spatially aligned to one another, the user does not perceive the marks 1502 and 1504 as being fused together; [0304]: alignment markers 1502 and 1504 are initially presented at the same positions of the left and right eye displays, though deformation may prompt perception of the positions as non-aligned. In some embodiments, the system may intentionally introduce a vertical offset between alignment markers 1502 and 1504. In such embodiments, the alignment of markers 1502 and 1504 may not necessarily be representative of the alignment of the left and right eye displays. For instance, although the left and right eye displays may exhibit relatively little or no misalignment, in these embodiments, the markers 1502 and 1504 presented by the system may exhibit a relatively high amount of misalignment. Doing so may serve to promote user engagement in the display alignment process). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply Vlaskamp’s knowledge of providing the left-eye and right-eye alignment markers on the selected positions on displayed imagery as taught and modify the system of Juenger because such a system uses user feedback to adjust content displayed through the system by compensating for any vertical alignment differences identified by the user, thus improving the user's comfort when viewing the HMD ([0160]).
Regarding claim 2, the combination of Juenger and Vlaskamp teaches the head-wearable device of claim 1, wherein: the head-wearable device further comprises one or more imaging devices (Juenger – fig. 1 camera(s) 120 and [0023]: sensors 114 are illustrated as including one or more cameras 120. In one or more implementations, the cameras 120 include at least a first camera mounted on a first side (e.g., the left side) of the housing 112 of computing device 102 that is configured to detect an alignment pattern in a left image of the digital content 106, and a second camera mounted on a second side (e.g., a right side) of the housing 112 of computing device 102 that is configured to detect an alignment pattern in a right image of the digital content 106), and the instructions for selecting one or both of (i) the selected point in time at which to present the realignment pattern via the head-wearable device and (ii) the selected location within the image at which the realignment pattern should be presented include: determining the selected location as a location at which the realignment pattern would be within peripheral vision of the user (Juenger – [0016]: an alignment pattern that is configured to enable automatic realignment of the stereoscopic digital content may be generated within the stereoscopic digital content such that the alignment pattern is within the field of view of the viewer; Vlaskamp – fig. 16 image 1602 shows markers 1502, 1504 and 1506 in the field of view of the user; Vlaskamp - [0303]: When a user wearing the HMD views the screens 1600a and 1600b simultaneously (e.g., with their left and right eyes respectively), the vertical alignment markers 1506a and 1506b may appear to the user as being fused together (e.g., as mark 1506). However, at least because the left-eye and right-eye horizontal alignment markers 1502 and 1504 are not spatially aligned to one another, the user does not perceive the marks 1502 and 1504 as being fused together).
Regarding claim 4, the combination of Juenger and Vlaskamp teaches the head-wearable device of claim 1, wherein: the head-wearable device further comprises one or more imaging devices, and the instructions for selecting one or both of (i) the selected point in time at which to present the realignment pattern via the head-wearable device and (ii) the selected location within the image at which the realignment pattern should be presented include: determining the selected point in time as a point in time during which the user's first eye or the user's second eye is blinking (alignment pattern is selected to be displayed at a point in time when a sudden movement by the viewer is detected; sudden movement by the viewer inherently includes rapid eye movements such as blinking and saccades; Juenger – [0032]: as the viewer moves their head, or moves around the room, the alignment of the left and right digital images of digital content 106 may become misaligned; Juenger – [0033]: Thus, in accordance with various implementations, alignment module 126 is configured to re-align the left and right images of the digital content 106 automatically, which is often referred to as a “binocular adjustment”. In order to perform the realignment, alignment module 126 causes alignment patterns 128 to be generated within the digital content 106. In some cases, the alignment module 126 may generate the alignment patterns 128 at periodic time intervals, such as when the computing device 102 is powered on, and every 30 seconds thereafter. Alternately, alignment module 126 may be configured to detect conditions which may be indicative of misalignment, such as a sudden movement by the viewer 108, and thus perform the alignment in response to this detection; Vlaskamp – [0169]: The wearable system 200 may also include an inward-facing imaging system 462 (shown in FIG. 4) which may track the eye movements of the user. The inward-facing imaging system may track either one eye's movements or both eyes' movements. The inward-facing imaging system 462 may be attached to the frame 230 and may be in electrical communication with the processing modules 260 or 270, which may process image information acquired by the inward-facing imaging system to determine, e.g., the pupil diameters or orientations of the eyes, eye movements or eye pose of the user 210; Vlaskamp – [0206]: The wearable system 400 may also include an inward-facing imaging system 466 (e.g., a digital camera), which observes the movements of the user, such as the eye movements and the facial movements; Vlaskamp – [0243]: eye tracking module 614 may include a blink detection module that provides a flag or other alert whenever a user blinks and a saccade detection module that provides a flag or other alert whenever a user's eye saccades (i.e., quickly shifts focus to another point); Vlaskamp – [0248]: Ocular event detection module 750 may receive other eye data from the eye tracking module 614 of FIG. 7A and may cause depth plane selection module 750 to delay some depth plane switches until an ocular event occurs. As an example, ocular event detection module 750 may cause depth plane selection module 750 to delay a planned depth plane switch until a user blink is detected; may receive data from a blink detection component in eye tracking module 614 that indicates when the user is currently blinking; and, in response, may cause depth plane selection module 750 to execute the planned depth plane switch during the blink event (such by causing module 750 to direct display 220 to execute the depth plane switch during the blink event). In at least some embodiments, the wearable system may be able to shift content onto a new depth plane during a blink event such that the user is unlikely to perceive the shift. As another example, ocular event detection module 750 may delay planned depth plane switches until an eye saccade is detected).
Regarding claim 6, the combination of Juenger and Vlaskamp teaches the head-wearable device of claim 1, wherein: the head-wearable device further comprises one or more imaging devices, and the instructions for selecting one or both of (i) the selected point in time at which to present the realignment pattern via the head-wearable device and (ii) the selected location within the image at which the realignment pattern should be presented include: determining the selected point in time as a point in time during which the user's first eye and the user's second eye are performing a saccade (alignment pattern is selected to be displayed at a point in time when a sudden movement by the viewer is detected; sudden movement by the viewer inherently includes rapid eye movements such as blinking and saccades; Juenger – [0032]: as the viewer moves their head, or moves around the room, the alignment of the left and right digital images of digital content 106 may become misaligned; Juenger – [0033]: Thus, in accordance with various implementations, alignment module 126 is configured to re-align the left and right images of the digital content 106 automatically, which is often referred to as a “binocular adjustment”. In order to perform the realignment, alignment module 126 causes alignment patterns 128 to be generated within the digital content 106. In some cases, the alignment module 126 may generate the alignment patterns 128 at periodic time intervals, such as when the computing device 102 is powered on, and every 30 seconds thereafter. Alternately, alignment module 126 may be configured to detect conditions which may be indicative of misalignment, such as a sudden movement by the viewer 108, and thus perform the alignment in response to this detection; Vlaskamp – [0169]: The wearable system 200 may also include an inward-facing imaging system 462 (shown in FIG. 4) which may track the eye movements of the user. The inward-facing imaging system may track either one eye's movements or both eyes' movements. The inward-facing imaging system 462 may be attached to the frame 230 and may be in electrical communication with the processing modules 260 or 270, which may process image information acquired by the inward-facing imaging system to determine, e.g., the pupil diameters or orientations of the eyes, eye movements or eye pose of the user 210; Vlaskamp – [0206]: The wearable system 400 may also include an inward-facing imaging system 466 (e.g., a digital camera), which observes the movements of the user, such as the eye movements and the facial movements; Vlaskamp – [0243]: eye tracking module 614 may include a blink detection module that provides a flag or other alert whenever a user blinks and a saccade detection module that provides a flag or other alert whenever a user's eye saccades (i.e., quickly shifts focus to another point); Vlaskamp – [0248]: Ocular event detection module 750 may receive other eye data from the eye tracking module 614 of FIG. 7A and may cause depth plane selection module 750 to delay some depth plane switches until an ocular event occurs. As an example, ocular event detection module 750 may cause depth plane selection module 750 to delay a planned depth plane switch until a user blink is detected; may receive data from a blink detection component in eye tracking module 614 that indicates when the user is currently blinking; and, in response, may cause depth plane selection module 750 to execute the planned depth plane switch during the blink event (such by causing module 750 to direct display 220 to execute the depth plane switch during the blink event). In at least some embodiments, the wearable system may be able to shift content onto a new depth plane during a blink event such that the user is unlikely to perceive the shift. As another example, ocular event detection module 750 may delay planned depth plane switches until an eye saccade is detected).
Regarding claim 7, the combination of Juenger and Vlaskamp teaches the head-wearable device of claim 1, wherein the instructions for selecting one or both of (i) the selected point in time at which to present the realignment pattern via the head-wearable device and (ii) the selected location within the image at which the realignment pattern should be presented include: determining the selected point in time as a point in time during which the user moves their head (Juenger – [0032]: as the viewer moves their head, or moves around the room, the alignment of the left and right digital images of digital content 106 may become misaligned; Juenger – [0033]: Thus, in accordance with various implementations, alignment module 126 is configured to re-align the left and right images of the digital content 106 automatically, which is often referred to as a “binocular adjustment”. In order to perform the realignment, alignment module 126 causes alignment patterns 128 to be generated within the digital content 106. In some cases, the alignment module 126 may generate the alignment patterns 128 at periodic time intervals, such as when the computing device 102 is powered on, and every 30 seconds thereafter. Alternately, alignment module 126 may be configured to detect conditions which may be indicative of misalignment, such as a sudden movement by the viewer 108, and thus perform the alignment in response to this detection; Vlaskamp – [0206]: The wearable system 400 may also include an inward-facing imaging system 466 (e.g., a digital camera), which observes the movements of the user, such as the eye movements and the facial movements … The wearable system 400 may also determine head pose (e.g., head position or head orientation) using sensors such as IMUs, accelerometers, gyroscopes, etc.).
Regarding claim 8, the combination of Juenger and Vlaskamp teaches the head-wearable device of claim 1, wherein the instructions for selecting one or both of (i) the selected point in time at which to present the realignment pattern via the head-wearable device and (ii) the selected location within the image at which the realignment pattern should be presented include: determining the point in time as a boot-up period (period when powering on the computer device 102) of the head-wearable device (Juenger - [0033]: alignment module 126 causes alignment patterns 128 to be generated within the digital content 106. In some cases, the alignment module 126 may generate the alignment patterns 128 at periodic time intervals, such as when the computing device 102 is powered on, and every 30 seconds thereafter).
Regarding claim 9, the combination of Juenger and Vlaskamp teaches the head-wearable device of claim 1, wherein the instructions for selecting one or both of (i) the selected point in time at which to present the realignment pattern via the head-wearable device and (ii) the selected location within the image at which the realignment pattern should be presented are executed in accordance with a determination that the image, as presented to the user's first and second eyes, satisfies misalignment criteria (Juenger – [0016]: Various environmental factors may cause the left and right images of the stereoscopic digital content to become misaligned while the viewer is using the computing device. Thus, an alignment pattern that is configured to enable automatic realignment of the stereoscopic digital content may be generated within the stereoscopic digital content such that the alignment pattern is within the field of view of the viewer. Generating the alignment pattern within the field of view of the viewer reduces the need to utilize additional display pixels (e.g., which are outside the field-of-view of the viewer) for the sole purpose of displaying the alignment pattern; Juenger – [0028]: The user experience manager module 104 is also illustrated as including an alignment module 126 that is configured to perform a “binocular alignment” of the digital content 106. When display device 116 is implemented as stereoscopic display 103, the alignment module 126 is configured to detect that the left and right images of the digital content 106 are misaligned. In response, the alignment module adjusts at least one of the left or right images of the digital content 106 in order to realign the digital content 106. Alternately, to realign the digital content 106, the alignment module 126 can guide a physical realignment of display device 116; Juenger – [0033]: Alternately, alignment module 126 may be configured to detect conditions which may be indicative of misalignment, such as a sudden movement by the viewer 108, and thus perform the alignment in response to this detection; Vlaskamp – [0302]: As shown in FIG. 16, the HMD may provide different respective alignment markers to the user's left and right eyes in order to demonstrate any left-right vertical misalignment. For example, the HMD may display screen 1600a to a user's left eye and may display screen 1600b to the user's right eye. Screen 1600a may include a left-eye horizontal alignment marker 1502 and a vertical alignment marker 1506, while screen 1600b may include a right-eye horizontal alignment marker 1504 and a vertical alignment marker 1506. In some embodiments, screen 1600a may not include the right-eye horizontal alignment marker 1504, and screen 1600b may not include the left-eye horizontal alignment marker 1502; Vlaskamp – [0304]: In at least some embodiments, alignment markers 1502 and 1504 are initially presented at the same positions of the left and right eye displays, though deformation may prompt perception of the positions as non-aligned. In some embodiments, the system may intentionally introduce a vertical offset between alignment markers 1502 and 1504. In such embodiments, the alignment of markers 1502 and 1504 may not necessarily be representative of the alignment of the left and right eye displays. For instance, although the left and right eye displays may exhibit relatively little or no misalignment, in these embodiments, the markers 1502 and 1504 presented by the system may exhibit a relatively high amount of misalignment. Doing so may serve to promote user engagement in the display alignment process. That is, some users may feel more compelled to actively participate in the display alignment process when the markers 1502 and 1504, as presented by the system, exhibit a relatively high degree of misalignment. In some examples, the vertical positions of the alignment markers 1502 and 1504 may be randomized, pseudorandomized, or quasi-randomized. For example, the system may randomly, pseudorandomly, or quasi-randomly select the vertical positions of the alignment markers 1502 and 1504 from a predetermined range of vertical positions. In some embodiments, the system may randomly, pseudorandomly, or quasi-randomly select the offset between the alignment markers 1502 and 1504 from a predetermined range of offsets. In some embodiments, the display system may be configured to randomly, pseudorandomly, or quasi-randomly select the offset based on pseudorandom number generators, such as a Mersenne Twister, Xorshift, and so on. In some embodiments, the pseudorandom number generators may generate the offset values directly. In some other embodiments, each offset may be associated with a number, or set of numbers, and the generation of a number by a pseudorandom number generators may be used to select the offset associated with that generated number or with the set to which that generated number belongs (e.g., each offset may be associated with one or more unique numbers or sets of numbers, a number may be generated using the pseudorandom generator, and the offset to be applied may be selected based on the correspondence between that generated number (or the set to which the number belongs) and the numbers (or sets of numbers) associated with the offsets); Vlaskamp – [0305]: The system may take the selected vertical positions of the markers 1502 and 1504 and/or the selected offset between the markers 1502 and 1504 into account when the user provides input to vertically align the markers 1502 and 1504. After the user provides input to vertically align the marks 1502 and 1504, the system may be able to determine the magnitude and direction of the left-right vertical misalignment based on the magnitude and direction of the user input. Preferably, adjustments to the vertical alignment is performed on only one display at a time).
Regarding claim 10, the combination of Juenger and Vlaskamp teaches the head-wearable device of claim 1, wherein the instructions for selecting one or both of (i) the selected point in time at which to present the realignment pattern via the head-wearable device and (ii) the selected location within the image at which the realignment pattern should be presented are executed at predetermined periods of time (every 30 seconds after the computing device is powered on is functionally analogous to predetermined periods of time; Juenger - [0033]: alignment module 126 causes alignment patterns 128 to be generated within the digital content 106. In some cases, the alignment module 126 may generate the alignment patterns 128 at periodic time intervals, such as when the computing device 102 is powered on, and every 30 seconds thereafter).
Regarding claim 11, the combination of Juenger and Vlaskamp teaches the head-wearable device of claim 1, wherein: the head-wearable device further comprises an eye-tracking camera, and the instructions for selecting one or both of (i) the selected point in time at which to present the realignment pattern via the head-wearable device and (ii) the selected location within the image at which the realignment pattern should be presented are based on data collected by the eye-tracking camera of the head-wearable device (alignment pattern is selected to be displayed when a sudden movement by the viewer is detected; sudden movement by the viewer inherently includes head movements or eye movements such as blinking and saccades; Juenger – [0032]: as the viewer moves their head, or moves around the room, the alignment of the left and right digital images of digital content 106 may become misaligned; Juenger – [0033]: Thus, in accordance with various implementations, alignment module 126 is configured to re-align the left and right images of the digital content 106 automatically, which is often referred to as a “binocular adjustment”. In order to perform the realignment, alignment module 126 causes alignment patterns 128 to be generated within the digital content 106. In some cases, the alignment module 126 may generate the alignment patterns 128 at periodic time intervals, such as when the computing device 102 is powered on, and every 30 seconds thereafter. Alternately, alignment module 126 may be configured to detect conditions which may be indicative of misalignment, such as a sudden movement by the viewer 108, and thus perform the alignment in response to this detection; Vlaskamp – [0169]: The wearable system 200 may also include an inward-facing imaging system 462 (shown in FIG. 4) which may track the eye movements of the user. The inward-facing imaging system may track either one eye's movements or both eyes' movements. The inward-facing imaging system 462 may be attached to the frame 230 and may be in electrical communication with the processing modules 260 or 270, which may process image information acquired by the inward-facing imaging system to determine, e.g., the pupil diameters or orientations of the eyes, eye movements or eye pose of the user 210; Vlaskamp – [0206]: The wearable system 400 may also include an inward-facing imaging system 466 (e.g., a digital camera), which observes the movements of the user, such as the eye movements and the facial movements; Vlaskamp – [0243]: eye tracking module 614 may include a blink detection module that provides a flag or other alert whenever a user blinks and a saccade detection module that provides a flag or other alert whenever a user's eye saccades (i.e., quickly shifts focus to another point); Vlaskamp – [0248]: Ocular event detection module 750 may receive other eye data from the eye tracking module 614 of FIG. 7A and may cause depth plane selection module 750 to delay some depth plane switches until an ocular event occurs. As an example, ocular event detection module 750 may cause depth plane selection module 750 to delay a planned depth plane switch until a user blink is detected; may receive data from a blink detection component in eye tracking module 614 that indicates when the user is currently blinking; and, in response, may cause depth plane selection module 750 to execute the planned depth plane switch during the blink event (such by causing module 750 to direct display 220 to execute the depth plane switch during the blink event). In at least some embodiments, the wearable system may be able to shift content onto a new depth plane during a blink event such that the user is unlikely to perceive the shift. As another example, ocular event detection module 750 may delay planned depth plane switches until an eye saccade is detected).
Regarding claim 12, the combination of Juenger and Vlaskamp teaches the head-wearable device of claim 1, wherein: the head-wearable device further comprises a disparity sensor (cameras 120, [0023] and [0034] - Juenger), and the instructions for modifying the presentation characteristics are based on an image of the realignment pattern (Juenger - [0018]: the digital content is realigned by adjusting at least one of the left and right images of the stereoscopic digital content based on the detected alignment patterns; Juenger - [0028]: the alignment module adjusts at least one of the left or right images of the digital content 106 in order to realign the digital content 106. Alternately, to realign the digital content 106, the alignment module 126 can guide a physical realignment of display device 116; Juenger - [0034]: After the alignment patterns 128 are generated, the cameras 120 are controlled to detect the alignment patterns 128 in each of the left and right images of the digital content 106. The location and position of the alignment patterns 128 may then be used to automatically realign the left and right images of the digital content 106; Juenger - [0046]: Alignment module 126 knows the position of the alignment patterns 128 within the digital content 106, and thus alignment module 126 can use the detected alignment patterns to calculate an amount of offset of the alignment pattern 128 in both of the left and right images of the digital content 106. Then, alignment module 126 uses the offset to adjust at least one of the left or right images of the digital content to perform the realignment; Juenger - [0053]: At 608, the stereoscopic digital content is realigned by adjusting at least one of the left and right images of the stereoscopic digital content based on the detected alignment patterns. For example, alignment module 126 adjusts at least one of the left and right images of the digital content 106 in order to align the images), wherein the image of the realignment pattern is captured by the disparity sensor of the head-wearable device (Juenger – [0029]: the alignment patterns 128 are configured such that they are detectable by the one or more cameras 120 of computing device 102; Juenger – [0034]: After the alignment patterns 128 are generated, the cameras 120 are controlled to detect the alignment patterns 128 in each of the left and right images of the digital content 106. The location and position of the alignment patterns 128 may then be used to automatically realign the left and right images of the digital content 106; Juenger – [0045]: after the alignment patterns 128 are generated, alignment module 126 controls cameras 120 to detect alignment patterns 128 in each of the left and right images of the digital content. For example, a first camera mounted on a left side of the housing 112 of computing device 102 can detect alignment patterns 128 in the left image of the digital content 106 rendered for the viewer's left eye, and a second camera mounted on a right side of the housing 112 can detect alignment patterns 128 in the right image of the digital content 106 rendered for the viewer's right eye. Alternately, rather than utilizing two cameras to detect the alignment patterns 128, a single camera 120 may utilize a prism in order to view the left and right images at the same time).
Regarding claim 13, the combination of Juenger and Vlaskamp teaches the head-wearable device of claim 1, wherein the head-wearable device is a pair of artificial-reality glasses (Juenger – [0019]: environment 100 includes a computing device 102 configured for use in augmented reality and/or virtual reality scenarios; Juenger – [0021]: the housing 112 is configured to be worn on the head of a viewer 108 (e.g., as a head-mounted display device 118), such as through configuration as goggles, glasses, contact lens, and so forth; Juenger – [0030]: FIG. 2 illustrates an example 200 of digital content that can be rendered for viewing in an augmented reality or virtual reality environment; Vlaskamp – [0007]: The augmented reality system comprises a head-mounted display (HMD) configured to present virtual content by outputting light to a user and at least one processor communicatively coupled to the HMD. The HMD comprises a left-eye display configured to present virtual content to the user's left eye and a right-eye display configured to present virtual content to the user's right eye).
Regarding claim 14, the combination of Juenger and Vlaskamp teaches the head-wearable device of claim 1, wherein the first and second image-projection systems each include at least one respective waveguide (Vlaskamp – [0031]: the head-mounted display comprises a first waveguide stack configured to pass light from the world into the left eye of the user and a second waveguide stack configured to pass light from the world into the right eye of the user and wherein each waveguide stack comprises a plurality of waveguides; Vlaskamp – [0066]: the left-eye display comprises a first waveguide stack configured to pass light from the world into the left eye of the user and the right-eye display comprises a second waveguide stack configured to pass light from the world into the right eye of the user, and wherein each waveguide stack comprises a plurality of waveguides).
Claims 15 and 17 are similar in scope to claims 1 and 7 respectively, and therefore the examiner provides similar rationale to reject these claims. Moreover, Juenger teaches a non-transitory computer-readable storage medium ([0055] and [0057]).
Claims 18 and 20 are similar in scope to claims 1 and 7 respectively, and therefore the examiner provides similar rationale to reject these claims. Moreover, the combination of Juenger and Vlaskamp teaches a handheld intermediary processing device (controller 460 or controller 1701 is functionally analogous to the handheld intermediary processing device; Vlaskamp – fig. 4 input device 466; Vlaskamp – fig. 17 shows user’s hand 1700 holding the controller 1701, [0307]: FIG. 17 is a perspective view of a user's hand 1700 and a controller 1701 including various input devices 1702, 1704, 1706. Any of the input devices 1702, 1704, 1706 may be configured to provide inputs regarding the vertical shift desired to align the alignment markers 1502 and 1504) configured to process data for a head- wearable device (Juenger - head-mounted display device 118, fig. 1 and [0021]; Vlaskamp – [0194]: A controller 460 controls the operation of the stacked waveguide assembly 480 and the image injection devices 420, 422, 424, 426, 428. The controller 460 includes programming (e.g., instructions in a non-transitory computer-readable medium) that regulates the timing and provision of image information to the waveguides 440b, 438b, 436b, 434b, 432b. In some embodiments, the controller 460 may be a single integral device, or a distributed system connected by wired or wireless communication channels. The controller 460 may be part of the processing modules 260 or 270 (illustrated in FIG. 2); Vlaskamp – [0207]: The wearable system 400 may include a user input device 466 by which the user may input commands to the controller 460 to interact with the wearable system 400. For example, the user input device 466 may include a trackpad, a touchscreen, a joystick, a multiple degree-of-freedom (DOF) controller, a capacitive sensing device, a game controller, a keyboard, a mouse, a directional pad (D-pad), a wand, a haptic device, a totem (e.g., functioning as a virtual user input device), and so forth. A multi-DOF controller may sense user input in some or all possible translations (e.g., left/right, forward/backward, or up/down) or rotations (e.g., yaw, pitch, or roll) of the controller. A multi-DOF controller which supports the translation movements may be referred to as a 3DOF while a multi-DOF controller which supports the translations and rotations may be referred to as 6DOF. In some cases, the user may use a finger (e.g., a thumb) to press or swipe on a touch-sensitive input device to provide input to the wearable system 400 (e.g., to provide user input to a user interface provided by the wearable system 400). The user input device 466 may be held by the user's hand during the use of the wearable system 400. The user input device 466 may be in wired or wireless communication with the wearable system 400; Vlaskamp – [0209]: The user input device 466 (shown in FIG. 4) may be an embodiment of a totem, which may include a trackpad, a touchpad, a trigger, a joystick, a trackball, a rocker or virtual switch, a mouse, a keyboard, a multi-degree-of-freedom controller, or another physical input device. A user may use the totem, alone or in combination with poses, to interact with the wearable system or other users), configured to be worn by a user (Juenger - [0021]: the housing 112 is configured to be worn on the head of a viewer 108 (e.g., as a head-mounted display device 118), such as through configuration as goggles, glasses, contact lens, and so forth), wherein the handheld intermediary processing device includes one or more programs including instructions for presenting an artificial reality environment at the head-wearable device (Vlaskamp – [0194]: A controller 460 controls the operation of the stacked waveguide assembly 480 and the image injection devices 420, 422, 424, 426, 428. The controller 460 includes programming (e.g., instructions in a non-transitory computer-readable medium) that regulates the timing and provision of image information to the waveguides 440b, 438b, 436b, 434b, 432b; Vlaskamp – [0305]: The system may take the selected vertical positions of the markers 1502 and 1504 and/or the selected offset between the markers 1502 and 1504 into account when the user provides input to vertically align the markers 1502 and 1504. After the user provides input to vertically align the marks 1502 and 1504, the system may be able to determine the magnitude and direction of the left-right vertical misalignment based on the magnitude and direction of the user input. Preferably, adjustments to the vertical alignment is performed on only one display at a time; Vlaskamp - [0311]: At block 1810, the wearable system may perform a display alignment flow, which may include blocks such as block 1812 and 1814. At block 1812, the HMD may provide unfused left-eye and right-eye alignment markers. As an example, the HMD may display alignment marker 1502 of FIGS. 15 and 16 on a left-eye display and may display alignment marker 1504 of FIGS. 15 and 16 on a right-eye display, as discussed herein. As described above, in some examples, the HMD may display one or both of alignment markers 1502 and 1504 at randomly-, pseudorandomly-, or quasi-randomly-selected vertical positions. At block 1814, the wearable system may receive user feedback on left-eye or right-eye display vertical alignment adjustments. In particular and as discussed in connection with FIGS. 15-17, the wearable system may receive user inputs that shift at least one of the alignment markers 1502 and 1504 until the markers are vertically aligned with each other from the perspective of the user. Blocks 1812 and 1814 may continue until the user exits the alignment process or accepts any vertical adjustments they have made to the alignment markers. As described above, in some examples, the wearable system may conduct one display alignment process per waveguide included in the HMD. In these examples, the wearable system may perform the operations associated with one or more of blocks 1810, 1812, and 1814 for each waveguide included in the HMD).
Claim(s) 3 is/are rejected under 35 U.S.C. 103 as being unpatentable over Juenger, in view of Vlaskamp, and further in view of Harada (US 2019/0105781).
Regarding claim 3, the combination of Juenger and Vlaskamp teaches the head-wearable device of claim 1, wherein: the head-wearable device further comprises one or more imaging devices (Juenger – fig. 1 camera(s) 120 and [0023]: sensors 114 are illustrated as including one or more cameras 120. In one or more implementations, the cameras 120 include at least a first camera mounted on a first side (e.g., the left side) of the housing 112 of computing device 102 that is configured to detect an alignment pattern in a left image of the digital content 106, and a second camera mounted on a second side (e.g., a right side) of the housing 112 of computing device 102 that is configured to detect an alignment pattern in a right image of the digital content 106).
The combination of Juenger and Vlaskamp does not explicitly teach the instructions for selecting one or both of (i) the selected point in time at which to present the realignment pattern via the head-wearable device and (ii) the selected location within the image at which the realignment pattern should be presented include: determining the selected location as a location at which the realignment pattern would be within a blind spot of the user.
Harada teaches the instructions for selecting one or both of (i) the selected point in time at which to present the realignment pattern via the head-wearable device and (ii) the selected location within the image at which the realignment pattern should be presented include: determining the selected location as a location at which the realignment pattern would be within a blind spot of the user ([0065]: The position where the calibration pattern 330 is installed is referred to as a “calibration position”. In the first embodiment, the calibration position at which the calibration pattern 330 is installed is a position outside a range of the field of view of the robot camera 170 in a state where the robot 100 performs work in the work area WKA, and a position that can be taken with the external camera 300). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply Harada’s knowledge of using a position outside the field of view (i.e., a blind spot) for providing calibration pattern as taught and modify the system of Juenger and Vlaskamp to use an external camera to capture the alignment pattern provided at a blind spot of the user because such a system can easily and accurately perform the calibration of the external camera ([0007]).
Claim(s) 5, 16 and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Juenger, in view of Vlaskamp, and further in view of Hong (US 2013/0278779).
Regarding claim 5, the combination of Juenger and Vlaskamp does not explicitly teach the head-wearable device of claim 1, wherein the instructions for selecting one or both of (i) the selected point in time at which to present the realignment pattern via the head-wearable device and (ii) the selected location within the image at which the realignment pattern should be presented include: determining the selected location as at least one location within the image at which the realignment pattern would blend in with other image content.
Hong teaches the head-wearable device of claim 1, wherein the instructions for selecting one or both of (i) the selected point in time at which to present the realignment pattern via the head-wearable device and (ii) the selected location within the image at which the realignment pattern should be presented include: determining the selected location as at least one location (locations where the calibration patterns becomes blended with the background of the image) within the image at which the realignment pattern would blend in with other image content ([0023]: While the calibration pattern 22 is clearly visible in the schematic representation of image 20', it is to be understood that the calibration pattern may become blended with the background. For example, the contrast of the calibration pattern 22 may be much lower than the contrast of the background due to the semi-transparent property of the screen 12). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply Hong’s knowledge of lowering the contrast of the calibration pattern so blend it with the background of the image and modify the system of Juenger and Vlaskamp because such a system temporally varies a calibration pattern during the capturing of an image sequence so that temporal correlation may be applied on the sequence of captured images, and thereby enabling the calibration pattern to be separated from the image of the background and features to be extracted from the calibration pattern. Such a robust feature detection enables automatic calibration of the projector-camera system including the semi-transparent screen ([0012]).
Claims 16 and 19 are similar in scope to claim 5, and therefore the examiner provides similar rationale to reject these claims.
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Xiong (US 2024/0046577) describes to capture, by a calibration camera, a calibration pattern displayed on a display of a video see-through AR system. The method further includes determining, based on each of one or more models of one or more components of the video see-through AR system, a modification to the image and generating, based on the modification to the image, a transformation map for the camera that captured the accessed image. The transformation map identifies frame-independent transformations to apply for rendering a scene based on one or more subsequent images captured by that camera.
Oyama et al. (US 2010/0091027) describes the HMD 201 causes the pattern image generating unit 217 to generate a pattern image for obtaining the amount of color misregistration. The HMD 201 displays the pattern image at a designated position (e.g., a peripheral position in the display area) corresponding to one eye on the image display unit 209. At this time, no image is displayed for the other eye, and the amount of color misregistration is obtained for each eye.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JWALANT B AMIN whose telephone number is (571)272-2455. The examiner can normally be reached Monday-Friday 10am - 630pm CST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Said Broome can be reached at 571-272-2931. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JWALANT AMIN/Primary Examiner, Art Unit 2612
Prosecution Insights