DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments, see the remarks, filed 09/22/2025, with respect to the amended claim(s) 1, 8, 17, and 20 have been fully considered and moot in view of new grounds of rejection by relying on the teachings of Kiemele et al. (US 20190065026 A1) in view of Ishikawa (US 20200275086 A1) and Matsui et al. (US 20080030461 A1).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-4 and 6-16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kiemele et al. (US 20190065026 A1) in view of Ishikawa (US 20200275086 A1).
Regarding claims 1 and 8, Kiemele discloses a first device (figs. 1 and 9, [0063-0064]), for carrying out a method, comprising:
a display (104 of fig. 1); and
one or more processors ([0063-0064] The logic machine may include one or more processors configured to execute software instructions as disclosed in figure 9, [0068])
wherein the first device in communication with an input device (104 and 110 of fig. 1) and wherein the first device is configured to perform:
presenting, via the display (104 of fig. 1, HMD display, [0014-0015]), a three-dimensional environment (100 of fig. 1, [0014] FIG. 1 shows an example use environment 100 in which a user 102 is engaged in a three-dimensional (3D) virtual reality experience executed on a head-mounted display (HMD) device 104);
presenting, via the display and in the three-dimensional environment (100 and 108 of fig. 1), a representation of the input device (110, 113, and 114 of figs. 2-5) at a first location in the three-dimensional environment ([0016-0017] The signals may include any suitable information enabling tracking of input device 110, such as output from one or more inertial measurement units (e.g., accelerometer, gyroscope, magnetometer) in the input device, a global positioning system (GPS) sensor in the input device, etc. In some examples, the signals may indicate the position (e.g., in the form of a three-dimensional coordinate) and/or orientation (e.g., in the form of a three-dimensional rotational coordinate) of input device 110), wherein the representation of the input device (113 and 114 of figs. 1-5) has a position and an orientation in the three-dimensional environment that is based on a position and an orientation of the input device in a physical environment ([0016] were user 102 to view input device 110 in the real, physical environment held at the location represented in FIG. 1, the display and body would appear substantially coextensive with display portion 113 and body portion 114 in the virtual reality experience. As such, and by potentially rendering display portion 113 and body portion 114 with substantially the same apparent size and angular orientation as their real-world counterparts, the portions may mimic or simulate the real-world appearance of input device 110; [0017] The signals may include any suitable information enabling tracking of input device 110, such as output from one or more inertial measurement units (e.g., accelerometer, gyroscope, magnetometer) in the input device, a global positioning system (GPS) sensor in the input device, etc. In some examples, the signals may indicate the position (e.g., in the form of a three-dimensional coordinate) and/or orientation (e.g., in the form of a three-dimensional rotational coordinate) of input device 110; [0037]);
while presenting the three-dimensional environment and presenting the representation of the input device at the first location in the three-dimensional environment (108, 110, 113, and 114 of figs. 1-5), detecting an input ([0015] User input received by the touch sensor may include touch, hover, and/or hand gesture input, for example, and may be sensed via any suitable touch sensing technology (e.g., capacitive, resistive, acoustic, optical). The user input may influence the rendering of visual content 106 in various ways, examples of which are depicted below with reference to FIGS. 2-6. Briefly, and without limitation, input device 110 may enable one or more of (1) drawing of visual content, which may be displayed on HMD device 104 as two-dimensional (2D) or 3D imagery, (2) editing and/or manipulation of visual content rendered on the HMD device, and (3) display and selection of user interface controls that affect visual content, among other potential functions; [0021] the sensor system of HMD device 104 may detect (e.g., via an outward-facing image sensor as disclosed in [0053-0054]) a hand gesture that effects scrolling of representation 204. Similarly, gaze, gestural, and/or other inputs may effect or cease the display itself of representation 204; [0054] to detect movements, such as gesture-based inputs or other movements performed by a wearer or by a person or physical object in the physical space) to capture one or more images ([0017] Alternatively or in addition to receiving tracking signals from input device 110, sensor system 116 may include one or more optical sensors for tracking the input device. For example, sensor system 116 may employ a visible light and/or depth camera to locate and segment image data corresponding to input device 110 from other image data. Additional detail regarding example hardware configurations of HMD device 104 is described below with reference to FIGS. 8 and 9; [0021] the sensor system of HMD device 104 may detect (e.g., via an outward-facing image sensor as disclosed in figures 8 and 9) a hand gesture that effects scrolling of representation 204. Similarly, gaze, gestural, and/or other inputs may effect or cease the display itself of representation 204; [0051-0054] image sensors for capturing images in the environment) from a) a perspective corresponding to the input device ([0035, 0036, and 0037]), and/or b) a perspective corresponding to the representation of the input device (113 and 114 of figs. 2-5, [0019 and 0037]); and
in response to detecting the input ([0015] user input, [0017, 0021, and 0022] tracking the user input as a hand gesture), capturing the one or more images ([0017, 0021, 0051-0054]) from a) the perspective corresponding to the input device ([0035-0037]), and/or b) the perspective corresponding to the representation of the input device (113 and 114 of figs. 1-5, [0019 and 0037]).
It is noted that Kiemele is silent about wherein: in accordance with a determination that the input device does not include an image capture device, the representation of the input device includes a representation of a virtual image capture device, and capturing the one or more images is from the perspective corresponding to the representation of the virtual image capture device.
Ishikawa teaches wherein: in accordance with a determination that the input device (201 and 211 of fig. 1C, [0076] and an example, 201 of fig. 9) does not include an image capture device (201 of fig. 9, does not have an image device),
the representation of the input device includes a representation of a virtual image capture device ([0042] the input controller 201 is an input apparatus used for the control of a virtual viewpoint, and receives an operation for changing camera parameters of a virtual camera. In addition, the input controller 201 detects the position and the orientation of the input controller 201 in the real space; [0050] and [0060] a virtual viewpoint image captured by the second virtual camera that is to be displayed on the display apparatus 203, [0062] a positional relationship between the input controller 201 and the display apparatus 203 will be described with reference to FIG. 4; [0097] In step S624, by transmitting the virtual viewpoint image captured by the second virtual camera that has been received in step S622, to the display apparatus 203, the display update unit 303 controls the display apparatus 203 to display the virtual viewpoint image), and
capturing the one or more images is from the perspective corresponding to the representation of the virtual image capture device ([0060-0061] a virtual viewpoint image captured by the second virtual camera is to be displayed, [0097] In step S624, by transmitting the virtual viewpoint image captured by the second virtual camera that has been received in step S622, to the display apparatus 203, the display update unit 303 controls the display apparatus 203 to display the virtual viewpoint image).
Taking the teachings of Kiemele and Ishikawa together as a whole, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the determination of the input device and the captured image by the virtual image capture device of Ishikawa into the display device of Kiemele to reduce or enlarge the displayed size of virtual content in response to the user moving the input device.
Regarding claim 2, Kiemele discloses the method of claim 1, Kiemele further discloses wherein: the three-dimensional environment is a virtual reality environment (108 of fig. 1), and the one or more images includes include one or more images of the virtual reality environment from b) the perspective corresponding to the representation of the input device in the three-dimensional environment (113 and 114 of figs. 1-5).
Regarding claim 3, Kiemele discloses the method of claim 1, Kiemele further discloses wherein: the three-dimensional environment is a virtual reality environment (108 of fig. 2), and the one or more images includes include one or more images of the physical environment from a) the perspective corresponding to the input device in the physical environment ([0020-0021] gestural).
Regarding claim 4, Kiemele discloses the method of claim 1, Kiemele further discloses wherein: the three-dimensional environment is an augmented reality environment ([0023-0024]), and the one or more images includes include one or more images of the augmented reality environment from a) the perspective corresponding to the input device in the physical environment, and b) the perspective corresponding to the representation of the input device in the three-dimensional environment ([0023-0024], [0036 and 0046]).
Regarding claim 6, Kiemele discloses the method of claim 1, Kiemele further discloses wherein the input device includes a position sensor and/or an orientation sensor ([0055-0056]).
Regarding claim 7, Kiemele discloses the method of claim 1, Kiemele further discloses wherein the orientation of the representation of the input device is based on orientation data from an orientation sensor of the input device ([0017] The signals may include any suitable information enabling tracking of input device 110, such as output from one or more inertial measurement units (e.g., accelerometer, gyroscope, magnetometer) in the input device, a global positioning system (GPS) sensor in the input device, etc. In some examples, the signals may indicate the position (e.g., in the form of a three-dimensional coordinate) and/or orientation (e.g., in the form of a three-dimensional rotational coordinate) of input device 110).
Regarding claim 9, Kiemele discloses the first device of claim 8, Kiemele further discloses wherein the position and the orientation of the representation of the input device in the three-dimensional environment is based on image data of the input device detected by image sensors of the first device (116 of fig. 1, 812A and 812B of fig. 8).
Regarding claim 10, Kiemele discloses the first device of claim 8, Kiemele further discloses wherein presenting the representation of the input device at the first location in the three-dimensional environment includes presenting a user interface including selectable modes of operating the representation of the input device, including a selfie mode ([0036]).
Regarding claim 11, Kiemele discloses the first device of claim 8, Kiemele further discloses wherein: the input is detected via the first device (116 of fig. 1, [0017] Alternatively or in addition to receiving tracking signals from input device 110, sensor system 116 may include one or more optical sensors for tracking the input device. For example, sensor system 116 may employ a visible light and/or depth camera to locate and segment image data corresponding to input device 110 from other image data. Additional detail regarding example hardware configurations of HMD device 104 is described below with reference to FIGS. 8 and 9).
Regarding claim 12, Kiemele discloses the first device of claim 8, Kiemele further discloses wherein: the input is detected via the input device before being detected at the first device ([0021] For example, representation 204 may be scrolled to change the visual content shown therein based on a gaze direction of a user).
Regarding claim 13, Kiemele discloses the first device of claim 8, Kiemele further discloses wherein the first device is configured to perform: capturing audio associated with the three-dimensional environment from a perspective of the first device in the three-dimensional environment, the audio associated with the one or more images ([0021] As another example, HMD device 104 may reproduce audio received, or output recorded, from input device 110 for implementations in which the HMD device includes an audio output device (e.g., speaker, headphones)).
Regarding claim 14, Kiemele discloses the first device of claim 8, Kiemele further discloses wherein the first device is configured to perform: capturing audio associated with the three-dimensional environment from a perspective of the representation of the input device in the three-dimensional environment, the audio associated with the one or more images ([0021] As another example, HMD device 104 may reproduce audio received, or output recorded, from input device 110 for implementations in which the HMD device includes an audio output device (e.g., speaker, headphones)).
Regarding claim 15, Kiemele discloses the first device of claim 8, Kiemele further discloses wherein: presenting, via the display and in the three-dimensional environment, the representation of the input device at the first location in the three-dimensional environment with the position and orientation in the three-dimensional environment that is based on the position and the orientation of the input device in the physical environment is performed in response to detecting a predefined interaction with the input device or the first device (110 of fig.1, [0016, 0030, and 0036]).
Regarding claim 16, Kiemele discloses the first device of claim 8, Kiemele further discloses wherein the input device is a mobile phone (110 of fig. 1).
Claim(s) 17-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kiemele et al. (US 20190065026 A1) in view of Matsui et al. (US 20080030461 A1).
Regarding claim 17, Kiemele further discloses a non-transitory computer readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by a processor ([0063-0064]), cause a first device that is in communication with a display and an input device (104 and 110 of fig. 1) to perform:
presenting, via the display (104 of fig. 1, HMD display, [0014-0015]), a three-dimensional environment (100 of fig. 1, [0014] FIG. 1 shows an example use environment 100 in which a user 102 is engaged in a three-dimensional (3D) virtual reality experience executed on a head-mounted display (HMD) device 104);
presenting, via the display and in the three-dimensional environment (100 and 108 of fig. 1), a representation of the input device (110, 113, and 114 of figs. 2-5) at a first location in the three-dimensional environment ([0016-0017] The signals may include any suitable information enabling tracking of input device 110, such as output from one or more inertial measurement units (e.g., accelerometer, gyroscope, magnetometer) in the input device, a global positioning system (GPS) sensor in the input device, etc. In some examples, the signals may indicate the position (e.g., in the form of a three-dimensional coordinate) and/or orientation (e.g., in the form of a three-dimensional rotational coordinate) of input device 110), wherein the representation of the input device (113 and 114 of figs. 1-5) has a position and an orientation in the three-dimensional environment that is based on a position and an orientation of the input device in a physical environment ([0016] were user 102 to view input device 110 in the real, physical environment held at the location represented in FIG. 1, the display and body would appear substantially coextensive with display portion 113 and body portion 114 in the virtual reality experience. As such, and by potentially rendering display portion 113 and body portion 114 with substantially the same apparent size and angular orientation as their real-world counterparts, the portions may mimic or simulate the real-world appearance of input device 110; [0017] The signals may include any suitable information enabling tracking of input device 110, such as output from one or more inertial measurement units (e.g., accelerometer, gyroscope, magnetometer) in the input device, a global positioning system (GPS) sensor in the input device, etc. In some examples, the signals may indicate the position (e.g., in the form of a three-dimensional coordinate) and/or orientation (e.g., in the form of a three-dimensional rotational coordinate) of input device 110);
while presenting the three-dimensional environment and presenting the representation of the input device at the first location in the three-dimensional environment (108, 110, 113, and 114 of figs. 1-5), detecting an input ([0015] User input received by the touch sensor may include touch, hover, and/or hand gesture input, for example, and may be sensed via any suitable touch sensing technology (e.g., capacitive, resistive, acoustic, optical). The user input may influence the rendering of visual content 106 in various ways, examples of which are depicted below with reference to FIGS. 2-6. Briefly, and without limitation, input device 110 may enable one or more of (1) drawing of visual content, which may be displayed on HMD device 104 as two-dimensional (2D) or 3D imagery, (2) editing and/or manipulation of visual content rendered on the HMD device, and (3) display and selection of user interface controls that affect visual content, among other potential functions; [0021] the sensor system of HMD device 104 may detect (e.g., via an outward-facing image sensor as disclosed in [0053-0054]) a hand gesture that effects scrolling of representation 204. Similarly, gaze, gestural, and/or other inputs may effect or cease the display itself of representation 204; [0054] to detect movements, such as gesture-based inputs or other movements performed by a wearer or by a person or physical object in the physical space) to capture one or more images ([0017] Alternatively or in addition to receiving tracking signals from input device 110, sensor system 116 may include one or more optical sensors for tracking the input device. For example, sensor system 116 may employ a visible light and/or depth camera to locate and segment image data corresponding to input device 110 from other image data. Additional detail regarding example hardware configurations of HMD device 104 is described below with reference to FIGS. 8 and 9; [0021] the sensor system of HMD device 104 may detect (e.g., via an outward-facing image sensor as disclosed in figures 8 and 9) a hand gesture that effects scrolling of representation 204. Similarly, gaze, gestural, and/or other inputs may effect or cease the display itself of representation 204; [0051-0054] image sensors for capturing images in the environment) from a) a perspective corresponding to the input device ([0035, 0036, and 0037]), and/or b) a perspective corresponding to the representation of the input device (113 and 114 of figs. 2-5, [0019 and 0037]); and
in response to detecting the input ([0015] user input, [0017, 0021, and 0022] tracking the user input as a hand gesture), capturing the one or more images ([0017, 0021, 0051-0054]) from a) the perspective corresponding to the input device ([0035-0037]), and/or b) the perspective corresponding to the representation of the input device (113 and 114 of figs. 1-5, [0019 and 0037]).
It is noted that Kiemele is silent about wherein capturing the one or more images comprises compositing image data of at least a portion of a capture of the physical environment and at least a portion of a capture of virtual content of the three-dimensional environment.
Matsui teaches wherein capturing the one or more images comprises compositing image data of at least a portion of a capture of the physical environment and at least a portion of a capture of virtual content of the three-dimensional environment (101 and 111 of fig. 1, [0078] With this arrangement, the observer can observe an image obtained by combining the real image and virtual image via the image display unit such as an HMD or the like, the image combining unit 111 can determine to which position in the image, which is displayed on a display unit 121b and includes the real image and the virtual image, the pointing position belongs; [0121] In step S105, the image combining unit 111 displays an MR image obtained by combining the real image stored in the real image memory and the virtual image stored in the virtual image memory on the display unit 121b. This processing is the same as that executed in the conventional MR presentation system).
Taking the teachings of Kiemele and Matsui together as a whole, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the combining unit (111 of fig. 1) of Matsui into the display device of Kiemele to provide the combined real image and virtual content to the display of the head mount display for the sense of immersion the observer experiences can be enhanced.
Regarding claim 18, Kiemele and Matsui disclose the non-transitory computer readable storage medium of claim 17, Kiemele further discloses wherein the input device includes an image capture device, and wherein capturing the one or more images of the environment from a) the perspective corresponding to the input device in the physical environment includes capturing the one or more images via the image capture device of the input device ([0036]).
Regarding claim 19, Kiemele and Matsui disclose the non-transitory computer readable storage medium of claim 17, Kiemele further discloses wherein: presenting, via the display and in the three-dimensional environment, the representation of the input device at the first location in the three-dimensional environment includes presenting the one or more images on the representation of the input device (502 of fig. 6, [0031-0034]).
Regarding claim 20 Kiemele and Matsui disclose the non-transitory computer readable storage medium of claim 17, Kiemele further discloses
the portion of the capture of the physical environment corresponding to first image data detected by image sensors of the input device (502 of fig. 5 and 600 of fig. 6, the input device capturing an image, [0035] Input device 110 may thus function analogously to a real-world camera or other touch-sensitive input devices that provide image capture functionality in the real-world, but acting to capture image data or other representations in virtual reality), and
the portion of the capture of virtual content of the three-dimension environment corresponding to second image data presented via the display (113 and 114 of fig. 5, [0037] the HMD device 104 may render the representation in display portion 113, in which case input device 110 may appear as a virtual window into the real, physical environment), and different from image data detected by the image sensors of the input device (fig. 6, [0035]),
a first portion of image data of the one or more images is processed at the input device ([0035]), and
a second portion of image data of the one or more images is processed at the first device ([0037] the HMD device 104 may render the representation in display portion 113, in which case input device 110 may appear as a virtual window into the real, physical environment).
Claim(s) 5 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kiemele et al. (US 20190065026 A1) in view of Ishikawa (US 20200275086 A1) as applied to claim 1, and further in view of Atlas et al. (US 11023035 B1).
Regarding claim 5, Kiemele and Ishikawa teach the method of claim 1, Kiemele and Ishikawa do not teach wherein: in accordance with a determination that the position and the orientation of the input device is a first position and a first orientation in the physical environment, the representation of the input device has a first position and a first orientation in the three-dimensional environment, and in accordance with a determination that the position and the orientation of the input device is a second position and a second orientation in the physical environment, different from the first position and the first orientation in the physical environment, the representation of the input device has a second position and a second orientation in the three-dimensional environment, different from the first position and the first orientation in the three-dimensional environment.
Atlas teaches in accordance with a determination that the position and the orientation of the input device is a first position and a first orientation in the physical environment (Col. 17, lines 11-20, As shown in FIG. 1B, in addition to or alternatively to image data captured via camera 138 of HMD 112C, input data from external cameras 102 may be used to track and detect particular motions, configurations, positions, and/or orientations of peripheral device 136 and/or hands and arms of users 110, such as hand 132 of user 110C, including movements of individual and/or combinations of digits (fingers, thumb) of the hand), the representation of the input device has a first position and a first orientation in the three-dimensional environment (136 of fig. 1A, Col. 26, lines 5-17), and
in accordance with a determination that the position and the orientation of the input device is a second position and a second orientation in the physical environment (Col. 17, lines 11-20, As shown in FIG. 1B, in addition to or alternatively to image data captured via camera 138 of HMD 112C, input data from external cameras 102 may be used to track and detect particular motions, configurations, positions, and/or orientations of peripheral device 136 and/or hands and arms of users 110, such as hand 132 of user 110C, including movements of individual and/or combinations of digits (fingers, thumb) of the hand), different from the first position and the first orientation in the physical environment, the representation of the input device has a second position and a second orientation in the three-dimensional environment, different from the first position and the first orientation in the three-dimensional environment (136 of fig. 1B, Col. 26, lines 35-47).
Taking the teachings of Kiemele, Ishikawa, and Atlas together as a whole, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the determined positions and orientations of Atlas into the detection of orientation and position of the input of Kiemele and Ishikawa for the display device reduces the size of virtual content in response to the user moving the input device toward the virtual surface.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TUNG T VO whose telephone number is (571)272-7340. The examiner can normally be reached Monday-Friday 6:30 AM - 5:00 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Brian Pendleton can be reached at 571-272-7527. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
TUNG T. VO
Primary Examiner
Art Unit 2425
/TUNG T VO/Primary Examiner, Art Unit 2425