DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 05/05/2026 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Response to Amendment
This is in response to applicant’s amendment/response filed on 08/12/2026, which has been entered and made of record. Claims 1, 5, and 10-11 have been amended. Claims 1-11 are pending in the application.
The amendment of specification is OK to Enter.
Response to Arguments
Applicant's arguments filed on 08/12/2026 have been fully considered but they are not persuasive or are rendered moot in view of the new grounds of rejection presented below (as necessitated by the amendment to claims 1 and 10-11).
The rejection of claim 11 under 35 USC 101 has been withdrawn after amendment.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-7 and 9-11 is/are rejected under 35 U.S.C. 103 as being unpatentable over U.S. PGPubs 2015/0312559 to Ueno et al. in view of U.S. PGPubs 2018/0120944 to Wang et al., further in view of U.S. PGPubs 2020/0183453 to Sano et al..
Regarding claim 1, Ueno et al. teach a method (par 0107), comprising: at a computer system that is in communication with a first display generation component, and one or more input devices (Fig 5, par 0092-0093):
displaying, via the first display generation component, a view of a three-dimensional environment (par 0090-0091, “The display device 1 has a function to three-dimensionally display virtual information, and to enable the user to operate the virtual information, in addition to the function to provide the user with a real scene as described above”), wherein the view of the three-dimensional environment concurrently includes first virtual content and a representation of a first portion of a physical environment (Fig 44, par 0300, “when the arrival point corresponds to Yokohama, the display device 6 may display a three-dimensional object of a gate of China town on a part of a real scene in a superimposed manner. For example, when the arrival point corresponds to Shizuoka, the display device 6 may display a three-dimensional object of Mt. Fuji that is seen from the arrival point on the background of the real scene in a superimposed manner”), the first portion of the physical environment includes a first physical surface (Fig 7, par 0107, “An image P1a is an image obtained by the imaging unit 40, that is, an image corresponding to a scene of the real space viewed by the right eye. In the image P1a, a table T1 and a hand H1 of the user appear. The display device 1 also acquires an image of the same scene imaged by the imaging unit 42, that is, an image corresponding to a scene of the real space viewed by the left eye”), and the first virtual content includes a first user interface object that is displayed at a position in the three-dimensional environment corresponding to a location of the first physical surface within the first portion of the physical environment (par 0109-0111, “At Step S1 illustrated in FIG. 7, the display device 1 composites the image P1a and the image P2a to generate an image P3a. The image P3a is an image displayed in the display unit 32a as an image for the right eye. At this time, the display device 1 determines the front and rear relationship between the real body existing in the image range of the imaging unit 40 and the three-dimensional object BL1 existing in the virtual space using the position of the right eye of the user as a reference point. Then, when the real body and the three-dimensional object BL1 overlap with each other, the display device 1 adjusts the overlapping such that one closer to the right eye of the user can be seen in front”);
while displaying the view of the three-dimensional environment, detecting a portion of a user at a first location within the first portion of the physical environment, wherein the first location is between the first physical surface and a viewpoint corresponding to the view of the three-dimensional environment (par 0006, “a display unit configured to three-dimensionally display a predetermined object, by displaying images respectively corresponding to both eyes of a user by being worn; a detection unit configured to detect displacement of a predetermined body in a display space of the object”, Fig 7, par 0102-0107, “The function provided by the display object control unit 26 includes a function to detect the operation to the three-dimensional object based on movement of the real body detected by the function of the detection processing unit 25, and change the three-dimensional object based on the detected operation ….An image P1a is an image obtained by the imaging unit 40, that is, an image corresponding to a scene of the real space viewed by the right eye. In the image P1a, a table T1 and a hand H1 of the user appear. The display device 1 also acquires an image of the same scene imaged by the imaging unit 42, that is, an image corresponding to a scene of the real space viewed by the left eye”);
in response to detecting the portion of the user at the first location within the first portion of the physical environment, ceasing to display a first portion of the first user interface object (Fig 7, par 0111-0114, “ the overlapping is adjusted for each range of a predetermined size, so that the hand H1 appears in front in a portion corresponding to the thumb of the hand H1, and the three-dimensional object BL1 appears in front in other portions, of a region where the hand H1 and the three-dimensional object BL1 overlap with each other, in the composited image P3a. Further, the three-dimensional object BL1 appears in front in a region where the table T1 and the three-dimensional object BL1 overlap with each other”); and
while displaying the view of the three-dimensional environment, detecting movement of the portion of the user from the first location to a second location within the first portion of the physical environment, wherein the second location is between the first physical surface and the viewpoint corresponding to the view of the three-dimensional environment (par 0104-0106, “The acting data 24c includes information related to how operation to the displayed three-dimensional object acts on the three-dimensional object. The acting data 24c is used for determining how to change the three-dimensional object when the operation to the displayed three-dimensional object is detected. The change referred to here includes movement, rotation, deformation, disappearance, and the like. The virtual space data 24d holds information related to a state of the three-dimensional object arranged in the virtual space. The state of the three-dimensional object includes, for example, a position, an attitude, a status of deformation, and the like”, Fig 7, par 0114-0115, “In the scene of Step S1 illustrated in FIG. 7, the user moves the hand H1 in the direction of an arrow A1. In this case, in the scene of Step S2 illustrated in FIG. 7, the image obtained by the imaging unit 40 is changed to an image P1b in which the position of the hand H1 is moved to the right. Further, the display device 1 determines that the movement of the hand H1 is operation to move the three-dimensional object to the right while holding the three-dimensional object, and moves the position of the three-dimensional object to the right in the virtual space according to the operation. The movement of the three-dimensional object in the virtual space is reflected in the virtual space data 24d”); and
in response to detecting the movement of the portion of the user from the first location to the second location, ceasing to display the second portion of the first user interface object so that the representation of the portion of the user is visible at a position that previously displayed the second portion of the first user interface object (Fig 7, par 0114-0115, “The display device 1 composites the image P1b and the image P2b to generate an image P3b for the right eye. The image P3b is an image that can be seen as if the user held the three-dimensional object BL1 with the hand H1 at a more right side on the table T1 than the image P1a. Similarly, the display device 1 generates a composite image for the left eye. The display device 1 then displays the composite images generated as described above in the display units 32a and 32b. As a result, the user can see the scene that is as if the own hand H1 had held the three-dimensional object BL1 and moved it to the right”).
But Ueno et al. keep silent for teaching the view of the three-dimensional environment concurrently includes first virtual content, second virtual content that is different from the first virtual content, and the second virtual content is displayed at a respective position in the three- dimensional environment that is different from the position corresponding to the location of the first physical surface, while displaying the view of the three-dimensional environment, detecting a portion of a user at a first location within the first portion of the physical environment; in response to detecting the portion of the user at the first location within the first portion of the physical environment, ceasing to display a first portion of the first user interface object while maintaining display of a second portion of the first user interface object so that a representation of the portion of the user is visible at a position that previously displayed the first portion of the first user interface object; and while displaying the view of the three-dimensional environment, detecting movement of the portion of the user from the first location to a second location within the first portion of the physical environment; and in response to detecting the movement of the portion of the user from the first location to the second location, restoring display of the first portion of the first user interface object and ceasing to display the second portion of the first user interface object so that the representation of the portion of the user is visible at a position that previously displayed the second portion of the first user interface object.
PNG
media_image1.png
310
508
media_image1.png
Greyscale
In related endeavor, Wang et al. teach the view of the three-dimensional environment concurrently includes first virtual content, second virtual content that is different from the first virtual content, and the second virtual content is displayed at a respective position in the three- dimensional environment that is different from the position corresponding to the location of the first physical surface (Fig 2, par 0045-0048, “In the example of FIG. 2, HMD device 18 displays virtual content in the form of a holographic motorcycle 238 and virtual targets in the forms of interactive holographic sliders 250, 254 and an interactive holographic push-button 258. In some examples one or more of the holographic motorcycle 238, interactive holographic sliders 250, 254 and interactive holographic push-button 258 may be displayed in a world-locked manner. The HMD device 18 may have a field of view, indicated by dotted lines 262, that defines a volume of space in which the user may view virtual content displayed by the device”….rendering 2 different contents 254 and 255 in field of user using HMD device), while displaying the view of the three-dimensional environment, detecting a portion of a user at a first location within the first portion of the physical environment (Fig 5, par 0057-0061, “based on determining that the user's point of gaze is at a gaze location within the target zone 304, and determining that the user's right hand 204 is at the hand location 514 within the designated tracking volume 504, the virtual affordance program 12 may cause HMD device 18 to display a virtual affordance at a landing location 530 corresponding to the interactive holographic slider 250”);
in response to detecting the portion of the user at the first location within the first portion of the physical environment, ceasing to display a first portion of the first user interface object while maintaining display of a second portion of the first user interface object so that a representation of the portion of the user is visible at a position that previously displayed the first portion of the first user interface object (Fig 3, par 0048, “To interact with a particular virtual target, initially the user 200 may direct his gaze toward the desired virtual target. With reference now to FIG. 3, this figure shows the couch 222 and interactive holographic sliders 250 and 254 as viewed by the user 200 through the HMD device 18”, Fig 5, par 0057-0063, “based on determining that the user's point of gaze is at a gaze location within the target zone 304, and determining that the user's right hand 204 is at the hand location 514 within the designated tracking volume 504, the virtual affordance program 12 may cause HMD device 18 to display a virtual affordance at a landing location 530 corresponding to the interactive holographic slider 250” …. In the example of FIG. 5, the landing location 530 is indicated by a plus sign located at the center of the interactive holographic slider 250. The landing location 530 may be spaced from the surface of the holographic slider 250 in the z-axis direction such that the holographic hand 520 is displayed to float just in front of the slider in the 3D model of the room 210”); and
while displaying the view of the three-dimensional environment, detecting movement of the portion of the user from the first location to a second location within the first portion of the physical environment (par 0061, “As described in more detail below, once the holographic hand 520 is displayed at the landing location, the user may now control movement of the holographic hand by moving the user's hand. The display of the holographic hand 520 is thereby modified in response to the movement of the user's hand”, par 0068-0069, “In response to the pinch gesture made by the user's right hand 204, the HMD device 18 may effect engagement between the holographic hand 520 and the holographic slider 250 such that movement of the holographic hand in an up/down y-axis direction by the user causes corresponding up/down y-axis movement of the slider along its axis of movement 540. For example and as shown in FIG. 7, the user 200 has moved his right hand 204 downwardly in a negative y-axis direction to cause the holographic hand 520 and holographic slider 250 to correspondingly move downwardly in the negative y-axis direction”); and
in response to detecting the movement of the portion of the user from the first location to the second location, restoring display of the first portion of the first user interface object and ceasing to display the second portion of the first user interface object so that the representation of the portion of the user is visible at a position that previously displayed the second portion of the first user interface object (par 0075-0078, “the virtual affordance program 12 may constrain the holographic hand 520 to move along an axis of the virtual target. In this manner, once the holographic hand 520 is engaged to control the virtual target at the landing location, movement of the corresponding hand of the user along the axis of the virtual target will cause corresponding movement of both the virtual affordance and the virtual target along that axis, while movement of the corresponding hand of the user along another axis different from the axis of the virtual target will not cause corresponding movement of the virtual affordance or the virtual target along that different axis”, Figs 9-10, par 0083-0084, “with reference to FIG. 10, the virtual affordance program 12 may determine that a left hand 208 of the user 200 is at a left hand location 516 within the designated tracking volume 504. As shown in FIG. 10, in this example the target zone 304 includes both of the interactive holographic sliders 250, 254. With reference to the descriptions above, based on determining that the user's point of gaze is at the gaze location/identifier 270 within the target zone 340, and determining that the left hand 208 of the user is at the left hand location 516 within the designated tracking volume 504, the virtual affordance program 12 may display a left hand virtual affordance 524 at a left hand landing location 534 (see FIG. 9) via the HMD device 18. The left hand landing location 534 is independent of both the user's gaze location within the target zone 304 and his left hand location within the designated tracking volume 504. As described above for the user's right hand 204, the HMD device 18 then may track movement of the user's left hand 208 and modify the display of the left hand virtual affordance 524 in response to the movement of the user's left hand. In this manner, the user 200 may use both left and right hands to interact with virtual targets in the manners described herein”).
It would have been obvious to a person of ordinary skill in the art at the time before the effective filing data of the claimed invention to modified Ueno et al. to include while displaying the view of the three-dimensional environment, detecting a portion of a user at a first location within the first portion of the physical environment; in response to detecting the portion of the user at the first location within the first portion of the physical environment, ceasing to display a first portion of the first user interface object while maintaining display of a second portion of the first user interface object so that a representation of the portion of the user is visible at a position that previously displayed the first portion of the first user interface object; and while displaying the view of the three-dimensional environment, detecting movement of the portion of the user from the first location to a second location within the first portion of the physical environment; and in response to detecting the movement of the portion of the user from the first location to the second location, restoring display of the first portion of the first user interface object and ceasing to display the second portion of the first user interface object so that the representation of the portion of the user is visible at a position that previously displayed the second portion of the first user interface object as taught by Wang et al. to manipulate a displayed virtual affordance to interact with one or more virtual objects by extending the user's arm and hand away from the user'body and using hand motions to control the virtual affordance and virtual object(s) to visually augment a view of a real world three dimensional environment through the display, such as by displaying virtual content that appears within the real world environment.
But Ueno et al. as modified by Wang et al. keep silent for teaching wherein at least a first portion of the representation of the portion of the user is visually obscured by the second virtual content when user interacts with first visual content, wherein display of the first portion of the representation of the portion of the user that was visually obscured by the second virtual content is restored when user don’t interact with first visual content.
PNG
media_image2.png
228
220
media_image2.png
Greyscale
In related endeavor, Sono et al. teach wherein at least a first portion of the representation of the portion of the user is visually obscured by the second virtual content when user interacts with first visual content, wherein display of the first portion of the representation of the portion of the user that was visually obscured by the second virtual content is restored when user don’t interact with first visual content (Figs 9-11, par 0053-0056, “By virtue of the above operation, as in a visual field image 134 illustrated in the lower figure of FIG. 10, the user visually recognizes the guide image 130 in the overlay image 132 such that the guide image partially overlaps his/her own fingers 76, and can visually recognize the layout image only on the writing surface 12 behind the fingers 76 without visually recognizing the virtual image of the layout image through your fingers 76. Even in this case, the user can visually recognize that the fingers 76 is positioned on the non-display processed image 102 displayed to coincide with the writing surface 12. Moreover, since the contents to be entered can be immediately understood from the guide image 130, the work can be efficiently performed. In addition, the guide image 130 may be displayed in a region that does not overlap the non-display region. However, as the guide image 130 overlaps the hand and is recognized to be present in front of the hand, the user can be caused to feel that the non-display processed image 102 including the layout image is present behind the hand, and this is preferable because the effect of reducing the discomfort is further enhanced. Moreover, as illustrated in FIG. 11, the entire guide image 130 may be displayed to overlap the non-display region. Since the guide image 130 is not overlaid on the layout image, it is easier to perceive that the layout image is positioned behind the real object 70, and the guide image 130 is positioned in front of the real object 70, thereby further eliminating the discomfort in the depth perception” ….considered a virtual element 100 as first virtual content and a virtual element 130 as second virtual content and 1. the virtual element 130 would virtual block the part of user’s hand when user hand 110 interacts with virtual element 100 (as shown in Fig 11) and 2. the virtual element 130 would not virtual block the part of user’s hand when user hand 110 don’t interact with virtual element 100 (when hand remove from the scene)).
It would have been obvious to a person of ordinary skill in the art at the time before the effective filing data of the claimed invention to modified Ueno et al. as modified by Wang et al. to include wherein at least a first portion of the representation of the portion of the user is visually obscured by the second virtual content when user interacts with first visual content, wherein display of the first portion of the representation of the portion of the user that was visually obscured by the second virtual content is restored when user don’t interact with first visual content as taught by and Sono et al. to disclose a method of controlling drawing of a virtual object (hereinafter referred to as an “AR object”) from a positional relationship of a real object in a depth direction with respect to the AR object.
Regarding claim 2, Ueno et al. as modified by Wang et al. and Sono et al. teach all the limitation of claim 1, and Wang et al. further teach including: while detecting the portion of the user at the first location within the first portion of the physical environment and maintaining display of the second portion of the first user interface object without displaying the first portion of the first user interface object, detecting a first input by the portion of the user that corresponds to a request to select the first user interface object; in response to detecting the first input by the portion of the user, performing a first operation corresponding to the first user interface object (Fig 5, par 0061-0067, “determined that the user's point of gaze is within the target zone 304 and the user's hand is within the designated tracking volume 504, the virtual affordance program 12 displays the holographic hand 520 at the landing location corresponding to the interactive holographic slider 250. As described in more detail below, once the holographic hand 520 is displayed at the landing location, the user may now control movement of the holographic hand by moving the user's hand. The display of the holographic hand 520 is thereby modified in response to the movement of the user's hand ….upon displaying the holographic hand 520 at the landing location 530 corresponding to the interactive holographic slider 250, the holographic hand may be engaged with the slider such that movement of user's right hand 204 in an up/down y-axis direction causes corresponding up/down y-axis movement of the slider along its axis of movement 540. In other examples, the holographic hand 520 may not be engaged with the holographic slider 250 upon the hand's initial display. In these examples, the user 200 may provide additional input to trigger such engagement. In some examples, the user 200 may perform a predetermined gesture that triggers engagement between the holographic hand and the holographic slider 250. In other examples, any other suitable form of user input may be utilized to trigger such engagement”); while detecting the portion of the user at the second location within the first portion of the physical environment and maintaining display of the first portion of the first user interface object without displaying the second portion of the first user interface object, detecting a second input by the portion of the user that corresponds to the request to select the first user interface object; and in response to detecting the second input by the portion of the user, performing a second operation corresponding to the first user interface object (Figs 8-10, par 0075-0084, “With reference to the descriptions above, based on determining that the user's point of gaze is at the gaze location/identifier 270 within the target zone 340, and determining that the left hand 208 of the user is at the left hand location 516 within the designated tracking volume 504, the virtual affordance program 12 may display a left hand virtual affordance 524 at a left hand landing location 534 (see FIG. 9) via the HMD device 18. The left hand landing location 534 is independent of both the user's gaze location within the target zone 304 and his left hand location within the designated tracking volume 504. As described above for the user's right hand 204, the HMD device 18 then may track movement of the user's left hand 208 and modify the display of the left hand virtual affordance 524 in response to the movement of the user's left hand. In this manner, the user 200 may use both left and right hands to interact with virtual targets in the manners described herein”).
Regarding claim 3, Ueno et al. as modified by Wang et al. and Sono et al. teach all the limitation of claim 1, and Wang et al. further teach including: while detecting the portion of the user at the first location within the first portion of the physical environment and maintaining display of the second portion of the first user interface object without displaying the first portion of the first user interface object, detecting a first input by the portion of the user that corresponds to a request to select the first portion of the first user interface object; in response to detecting the first input by the portion of the user, performing a first operation corresponding to the first portion of the first user interface object (Fig 5, par 0061-0067, “determined that the user's point of gaze is within the target zone 304 and the user's hand is within the designated tracking volume 504, the virtual affordance program 12 displays the holographic hand 520 at the landing location corresponding to the interactive holographic slider 250. As described in more detail below, once the holographic hand 520 is displayed at the landing location, the user may now control movement of the holographic hand by moving the user's hand. The display of the holographic hand 520 is thereby modified in response to the movement of the user's hand ….upon displaying the holographic hand 520 at the landing location 530 corresponding to the interactive holographic slider 250, the holographic hand may be engaged with the slider such that movement of user's right hand 204 in an up/down y-axis direction causes corresponding up/down y-axis movement of the slider along its axis of movement 540. In other examples, the holographic hand 520 may not be engaged with the holographic slider 250 upon the hand's initial display. In these examples, the user 200 may provide additional input to trigger such engagement. In some examples, the user 200 may perform a predetermined gesture that triggers engagement between the holographic hand and the holographic slider 250. In other examples, any other suitable form of user input may be utilized to trigger such engagement”); while detecting the portion of the user at the second location within the first portion of the physical environment and maintaining display of the first portion of the first user interface object without displaying the second portion of the first user interface object, detecting a second input by the portion of the user that corresponds to the request to select the second portion of the first user interface object; and in response to detecting the second input by the portion of the user, performing a second operation corresponding to the second portion of the first user interface object, wherein the second operation is different from the first operation (Figs 8-10, par 0075-0084, “With reference to the descriptions above, based on determining that the user's point of gaze is at the gaze location/identifier 270 within the target zone 340, and determining that the left hand 208 of the user is at the left hand location 516 within the designated tracking volume 504, the virtual affordance program 12 may display a left hand virtual affordance 524 at a left hand landing location 534 (see FIG. 9) via the HMD device 18. The left hand landing location 534 is independent of both the user's gaze location within the target zone 304 and his left hand location within the designated tracking volume 504. As described above for the user's right hand 204, the HMD device 18 then may track movement of the user's left hand 208 and modify the display of the left hand virtual affordance 524 in response to the movement of the user's left hand. In this manner, the user 200 may use both left and right hands to interact with virtual targets in the manners described herein”).
Regarding claim 4, Ueno et al. as modified by Wang et al. and Sono et al. teach all the limitation of claim 1, and further teach wherein the first virtual content includes a second user interface object that is displayed at a position in the three-dimensional environment corresponding to the location of the first physical surface within the first portion of the physical environment (Ueno et al.: Fig 7, par 0102-0107, “The function provided by the display object control unit 26 includes a function to detect the operation to the three-dimensional object based on movement of the real body detected by the function of the detection processing unit 25, and change the three-dimensional object based on the detected operation ….An image P1a is an image obtained by the imaging unit 40, that is, an image corresponding to a scene of the real space viewed by the right eye. In the image P1a, a table T1 and a hand H1 of the user appear. The display device 1 also acquires an image of the same scene imaged by the imaging unit 42, that is, an image corresponding to a scene of the real space viewed by the left eye”, Wang et al.: Fig 5, par 0057-0061, “based on determining that the user's point of gaze is at a gaze location within the target zone 304, and determining that the user's right hand 204 is at the hand location 514 within the designated tracking volume 504, the virtual affordance program 12 may cause HMD device 18 to display a virtual affordance at a landing location 530 corresponding to the interactive holographic slider 250”), and the method includes: in response to detecting the portion of the user at the first location within the first portion of the physical environment, ceasing to display a first portion of the second user interface object while maintaining display of a second portion of the second user interface object so that the representation of the portion of the user is visible at a position that previously displayed the first portion of the second user interface object (Wang et al.: Fig 3, par 0048, “To interact with a particular virtual target, initially the user 200 may direct his gaze toward the desired virtual target. With reference now to FIG. 3, this figure shows the couch 222 and interactive holographic sliders 250 and 254 as viewed by the user 200 through the HMD device 18”, Fig 5, par 0057-0063, “based on determining that the user's point of gaze is at a gaze location within the target zone 304, and determining that the user's right hand 204 is at the hand location 514 within the designated tracking volume 504, the virtual affordance program 12 may cause HMD device 18 to display a virtual affordance at a landing location 530 corresponding to the interactive holographic slider 250” …. In the example of FIG. 5, the landing location 530 is indicated by a plus sign located at the center of the interactive holographic slider 250. The landing location 530 may be spaced from the surface of the holographic slider 250 in the z-axis direction such that the holographic hand 520 is displayed to float just in front of the slider in the 3D model of the room 210”).
Regarding claim 5, Ueno et al. as modified by Wang et al. and Sono et al. teach all the limitation of claim 4, and Wang et al. further teach including: while detecting the portion of the user at the first location within the first portion of the physical environment and maintaining display of the second portion of the first user interface object and the second user interface object without displaying the first portion of the first user interface object and without displaying the first portion of the second user interface object, detecting a third input by the portion of the user that corresponds to a request to select the first user interface object; and in response to detecting the third input by the portion of the user, performing a third operation corresponding to the first user interface object, without performing a fourth operation corresponding to the second user interface object (Figs 8-10, par 0075-0084, “With reference to the descriptions above, based on determining that the user's point of gaze is at the gaze location/identifier 270 within the target zone 340, and determining that the left hand 208 of the user is at the left hand location 516 within the designated tracking volume 504, the virtual affordance program 12 may display a left hand virtual affordance 524 at a left hand landing location 534 (see FIG. 9) via the HMD device 18. The left hand landing location 534 is independent of both the user's gaze location within the target zone 304 and his left hand location within the designated tracking volume 504. As described above for the user's right hand 204, the HMD device 18 then may track movement of the user's left hand 208 and modify the display of the left hand virtual affordance 524 in response to the movement of the user's left hand. In this manner, the user 200 may use both left and right hands to interact with virtual targets in the manners described herein”).
Regarding claim 6, Ueno et al. as modified by Wang et al. and Sono et al. teach all the limitation of claim 4, and Wang et al. further teach including: in response to detecting the movement of the portion of the user from the first location to the second location, restoring display of the first portion of the second user interface object and ceasing to display the second portion of the second user interface object so that the representation of the portion of the user is visible at a position that previously displayed the second portion of the second user interface object (Figs 8-10, par 0075-0084, “With reference to the descriptions above, based on determining that the user's point of gaze is at the gaze location/identifier 270 within the target zone 340, and determining that the left hand 208 of the user is at the left hand location 516 within the designated tracking volume 504, the virtual affordance program 12 may display a left hand virtual affordance 524 at a left hand landing location 534 (see FIG. 9) via the HMD device 18. The left hand landing location 534 is independent of both the user's gaze location within the target zone 304 and his left hand location within the designated tracking volume 504. As described above for the user's right hand 204, the HMD device 18 then may track movement of the user's left hand 208 and modify the display of the left hand virtual affordance 524 in response to the movement of the user's left hand. In this manner, the user 200 may use both left and right hands to interact with virtual targets in the manners described herein”).
Regarding claim 7, Ueno et al. as modified by Wang et al. and Sono et al. teach all the limitation of claim 4, and Wang et al. further teach including: in response to detecting the movement of the portion of the user from the first location to the second location, maintaining display of the second portion of the second user interface object without restoring display of the first portion of the second user interface object so that the representation of the portion of the user is visible at the position that previously displayed the first portion of the second user interface object (Figs 8-10, par 0075-0084, “With reference to the descriptions above, based on determining that the user's point of gaze is at the gaze location/identifier 270 within the target zone 340, and determining that the left hand 208 of the user is at the left hand location 516 within the designated tracking volume 504, the virtual affordance program 12 may display a left hand virtual affordance 524 at a left hand landing location 534 (see FIG. 9) via the HMD device 18. The left hand landing location 534 is independent of both the user's gaze location within the target zone 304 and his left hand location within the designated tracking volume 504. As described above for the user's right hand 204, the HMD device 18 then may track movement of the user's left hand 208 and modify the display of the left hand virtual affordance 524 in response to the movement of the user's left hand. In this manner, the user 200 may use both left and right hands to interact with virtual targets in the manners described herein”).
Regarding claim 9, Ueno et al. as modified by Wang et al. and Sono et al. teach all the limitation of claim 1, and Ueno et al. further teach wherein the first user interface object is a virtual keyboard that includes at least a first key and a second key different from the first key, and the first portion of the first user interface object corresponds to the first key and the second portion of the first user interface object corresponds to the second key (par 0281, “to allow the user to operate a three-dimensionally displayed piano or keyboard with fingers to enjoy playing the instrument, in conjunction with an instrument play program. The control program 24e may be configured to, when a three-dimensionally displayed portion having an arbitrary shape is cut by a hand, cause an image of a cut surface to be projected from the projector 34, in conjunction with a data display program”, Figs 45-46, par 0310, “FIG. 45 is a diagram conceptually illustrating operation screens arranged around the user. FIG. 46 is a diagram illustrating one of examples of changing the three-dimensional object in conjunction with the change of a direction. In the example illustrated in FIGS. 45 and 46, the virtual space data 24d holds information related to states of three-dimensional objects of operation screens OS1 to OS3, and the object data 24b holds information related to shapes and properties of the three-dimensional objects of the operation screens OS1 to OS3”).
Regarding claim 10, Ueno et al. teach a computer system, comprising: a first display generation component; one or more input devices; one or more processors; and memory storing one or more programs, wherein the one or more programs are configured to be executed by the one or more processors, the one or more programs including instructions for (Fig 6, par 0093, par 0100-0101). The remaining limitations of the claim are similar in scope to claim 1 and rejected under the same rationale.
Regarding claim 11, Ueno et al. teach a computer readable storage medium storing one or more programs, the one or more programs comprising instructions that, when executed by a computer system that includes a first display generation component, and one or more input devices, cause the computer system to (par 0100-0101, par 0285). The remaining limitations of the claim are similar in scope to claim 1 and rejected under the same rationale.
Claim(s) 8 is/are rejected under 35 U.S.C. 103 as being unpatentable over U.S. PGPubs 2015/0312559 to Ueno et al. in view of U.S. PGPubs 2018/0120944 to Wang et al., further in view of U.S. PGPubs 2020/0183453 to Sano et al., further in view of US 2022/0229534 to Terre et al.
Regarding claim 8, Ueno et al. as modified by Wang et al. and Sono et al. teach all the limitation of claim 1, but keep silent for teaching including: in response to detecting the portion of the user at the first location within the first portion of the physical environment, displaying a simulated shadow of the portion of the user at a third position in the view of the three-dimensional environment that is offset from the position that previously displayed the first portion of the first user interface object; and in response to detecting the movement of the portion of the user from the first location to the second location, displaying the simulated shadow of the portion of the user at a fourth position in the view of the three-dimensional environment that is offset from the position that previously displayed the second portion of the first user interface object.
In related endeavor, Terre et al. teach including: in response to detecting the portion of the user at the first location within the first portion of the physical environment, displaying a simulated shadow of the portion of the user at a third position in the view of the three-dimensional environment that is offset from the position that previously displayed the first portion of the first user interface object; and in response to detecting the movement of the portion of the user from the first location to the second location, displaying the simulated shadow of the portion of the user at a fourth position in the view of the three-dimensional environment that is offset from the position that previously displayed the second portion of the first user interface object (Figs 22-24, par 0270-0271, “Once it is determined that the user 2220 may potentially wish to kinesthetically select one of the virtual objects in the group of virtual objects 2212, the group of virtual objects 2212 are highlighted in response to the initial kinesics input of the user 2220 tending toward the first virtual region 2214. In the example illustrated in FIG. 22, the visual appearance of each virtual object of the group of virtual objects 2212 in the first virtual region 2214 are shown to be emphasized, or otherwise differentiated, from the second subset of dispersed virtual objects 2216 in the second virtual region 2218 due to an added shadow around the frame of the virtual objects 2212”, par 0275-0276, “While the initial kinesics input and the refined kinesics input illustrated in FIG. 22 and FIG. 23 both depict a hand gesture of the user, it is to be understood that each of the initial kinesics input and the refined kinesics input may include any combination of gesture-based interactions including hand gestures and/or eye-movements. With reference to FIG. 23, the user 2310 is shown to express a specific interest in a particular virtual object 2312 from among the previously highlighted group of virtual objects within the first virtual region 2314 via a refined kinesics input. At the point in time illustrated in FIG. 23, the refined kinesics input of the user 2310 is more clearly tending toward the particular virtual object 2312 rather than the other virtual objects 2316 within the first virtual region 2314. Thus, a degree of certainty related to the refined kinesics input illustrated in FIG. 23 is greater than a degree of certainty related to the initial kinesics input illustrated in FIG. 22, and the refined kinesics input more specifically indicates that the user 2310 may wish to make a kinesics selection of a particular virtual object 2312. In response to the refined kinesics input, subsequent feedback may be provided to the user 2310 by unhighlighting the other virtual objects 2316 within the first virtual region 2314 in the periphery of the particular virtual object 2312 until only the particular virtual object 2312 determined by the at least one processor to be intended for selection beyond a predetermined degree of certainty remains highlighted based on the user's refined motions“).
It would have been obvious to a person of ordinary skill in the art at the time before the effective filing data of the claimed invention to modified Ueno et al. as modified by Wang et al. and Sono et al. to include including: in response to detecting the portion of the user at the first location within the first portion of the physical environment, displaying a simulated shadow of the portion of the user at a third position in the view of the three-dimensional environment that is offset from the position that previously displayed the first portion of the first user interface object; and in response to detecting the movement of the portion of the user from the first location to the second location, displaying the simulated shadow of the portion of the user at a fourth position in the view of the three-dimensional environment that is offset from the position that previously displayed the second portion of the first user interface object as taught by Terre et al. to provide a new approach for solving the productivity dilemma, one that uses extended reality (XR) to provide a mobile environment that enables users to experience the comfort of a stationary workspace anywhere they want by providing virtual desktop-like screens.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Jin Ge whose telephone number is (571)272-5556. The examiner can normally be reached 8:00 to 5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jason Chan can be reached at (571)272-3022. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
JIN . GE
Examiner
Art Unit 2619
/JIN GE/Primary Examiner, Art Unit 2619
