Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3-4, 6-10, 12-13 and 16-22 are rejected under 35 U.S.C. 103 as being unpatentable over Lacey et al. (US 20190362557 A1), hereinafter Lacey, in view of Tang et al. (US 20200226814 A1), hereinafter Tang and Connellan et al. (US 2019/0113966 A1), hereinafter Connellan .
Regarding Claim 1, Lacey teaches:
An information processing apparatus (See FIG. 2B: 200) comprising:
one or more processors and/or circuitry (260 or 128) (See paragraphs [0101], [0102] and FIG. 2B) configured to:
(1) execute display control processing of displaying a plurality of virtual objects (virtual objects 1210, 1220, 1230, 1242, and 1244) such that the plurality of virtual objects are arranged in a three-dimensional space (See paragraph [0177)) that isa visual field of a user (1250) (See FIG. 12B) (See paragraph [0172));
(2) execute first acquisition processing of acquiring information of a position of an operating body at a position of a hand of the user (See paragraph [0364], lines 15-19: the tracked controller or tracked hand targeting vector);
(3) execute second acquisition processing of acquiring information of a line-of-sight position of the user (See paragraph [0128], last four lines); and
(4) execute control processing of (a) switching to a first operation mode, in which the user performs an operation on a virtual object selected from the plurality of virtual objects by directly touching the virtual object with the operating body (See paragraph [0428] and FIG. 53: the first operation mode is a hand operation mode that enables the user to perform an operation on the first virtual object by directly touching and selecting the first virtual object with the user's hand using the convergence point 5301), in a case where a distance between the position of the operating body and the line-of-sight position (See paragraph [0082]: convergence between the head and hand results in a first operation mode) is smaller than a predetermined threshold (See paragraph [0084]; See paragraph [0364], lines 1-19), (b) switching to a second operation mode, in which the user performs an operation on the virtual object selected from the plurality of virtual objects by pointing to the virtual object with a distal end portion of a display item that extends from a position of the operating body(See paragraph [0428] and FIG. 53: the second operation mode isa ray operation mode that enables the user to perform an operation on the second virtual object by pointing a distal end portion of a ray to and selecting the second virtual object using the convergence point 5307), in a case where the distance between the position of the operating body and the line-of-sight position (See paragraph [0082]: divergence between the head and hand result in a second operation mode) is larger than the predetermined threshold (See paragraph [0373]; See paragraph [0414]).
Lacey does not explicitly teach:
the display item is beam shaped, and
(c) even when the second operation mode is set, switching to the first operation mode in a case where it is determined that the user is not in a predetermined posture in which the operating body is held in front of the user.
Tang discloses the display item is an item of a beam shape that extends from the position of the operating body (See Tang, paragraph [0015], lines 1-4).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the information processing apparatus of Lacey with the teaching of Tang so that would allow the user to directly target and select the virtual object using a natural interaction methodology as suggested by Tang (See paragraph [0055]).
Connellan discloses even when the second operation mode is set (e.g. 1520a in the FOV so the mode is set to a higher sensitivity) (Fig. 15, Pars. 146, 148, 153-154), switching to the first operation mode (e.g. 1520b not in the FOV so the mode is set to a lower sensitivity) in a case where it is determined that the user is not in a predetermined posture in which the operating body is held in front of the user (Fig. 15, Pars. 148, 153-154, see also Par. 155).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified Lacey with the teaching of Connellan to improve tracking in VR/AR system as suggested by Connellan (Par. 146).
Regarding Claim 9, see claim 1 rejection and motivation above. Lacey further teaches:
An information processing method (See the method depicted in FIG. 53, as well as the steps described below) comprising:
displaying a plurality of virtual objects (virtual objects 1210, 1220, 1230, 1242, and 1244) such that the plurality of virtual objects are arranged in a three-dimensional space (See paragraph [0177]) that is a visual field of a user (1250) (See FIG. 12B) (See paragraph [0172));
acquiring information of a position of an operating body at a position of a hand of the user (See paragraph [0364], lines 15-19: the tracked controller or tracked hand targeting vector);
acquiring information of a line-of-sight position of the user (See paragraph [0128], last four lines); and
setting a first operation mode in a case where a distance between the position of the operating body and the line-of-sight position (See paragraph [0082]: convergence between the head and hand results in a first operation mode) is smaller than a predetermined threshold (See paragraph [0084]; See paragraph [0364], lines 1-19);
setting a second operation mode in a case where the distance between the position of the operating body and the line-of-sight position (See paragraph [0082]: divergence between the head and hand result in a second operation mode) is larger than the predetermined threshold (See paragraph [0373]; See paragraph [0414]); and
and which is indicated by the item in the second operation mode (See paragraph [0428] and FIG. 53: an indicating a direction to which the operating body is directed (palm-to-fingertip ray cast) is determined, and the virtual object indicated by palm-to-fingertip raycast is selected in the second operation mode) (See also FIG. 46E, showing a selection of the virtual object in the second operation mode (Totem Projected Raycast /Hand Feature Point RayCast)).
Regarding Claim 10, see claim 1 rejection and motivation above. Lacey further teaches:
A non-transitory computer-readable storage medium that stores a program for causing a computer (See paragraph [0508)]) to execute an information processing method (See the method depicted in FIG. 53, as well as the steps described below), the method comprising:
displaying a plurality of virtual objects (virtual objects 1210, 1220, 1230, 1242, and 1244) such that the plurality of virtual objects are such that the virtual objects are arranged in a three-dimensional space (See paragraph [0177]) that is a visual field of a user (1250) (See FIG. 12B) (See paragraph [0172));
acquiring information of a position of an operating body at a position of a hand of the user (See paragraph [0364], lines 15-19: the tracked controller or tracked hand targeting vector);
acquiring information of a line-of-sight position of the user (See paragraph [0128], last four lines); and
setting a first operation mode in a case where a distance between the position of the operating body and the line-of-sight position (See paragraph [0082]: convergence between the head and hand results in a first operation mode) is smaller than a predetermined threshold (See paragraph [0084]; See paragraph [0364], lines 1-19);
setting a second operation mode in a case where the distance between the position of the operating body and the line-of-sight position (See paragraph [0082]: divergence between the head and hand result in a second operation mode) is larger than the predetermined threshold (See paragraph [0373]; See paragraph [0414]).
Regarding Claim 3, Lacey in view of Tang and Connellan teaches all of the elements of the claimed invention, as stated above. Furthermore, Lacey teaches:
The information processing apparatus according to claim 1, wherein in the control processing, (1) the first operation mode is set in a case where a state where the distance between the position of the operating body and the line-of-sight position is smaller than the predetermined threshold continues fora predetermined time, and (2) the second operation mode is set in a case where a state where the distance between the position of the operating body and the line-of-sight position is larger than the predetermined threshold continues for the predetermined time (See paragraph [0394]: transmodal input fusion techniques provide for dynamic coupling of sensor input, e.g., identifying sensor inputs that have converged temporally (e.g., for a fixation or dwell time); See paragraph [0415]).
Regarding Claim 4, Lacey in view of Tang and Connellan teaches all of the elements of the claimed invention, as stated above. Furthermore, Tang teaches:
In the control processing, the second operation mode is set in a case where none of a plurality of virtual objects are arranged within a predetermined range from the position of the operating body (See paragraph [0055)]). See claim 1 motivation above.
Regarding Claim 6, see claim 4 rejection and motivation above. Lacey in view of Tang and Connellan teaches all of the elements of the claimed invention, as stated above. Furthermore, Lacey in view of Tang and Connellan teaches:
In the control processing, control is performed not to display the display item in a case where the operating body is not located in front of the user (See Tang, paragraph [0040] and FIG. 4B: since the targeting ray 470 is case when the operating body is located in the field of view of the head-mounted display 425, the targeting ray 470 is therefore controlled not to be displayed in a case where the operating body is not located in front of the user (e.g., at the user’s side, out of the field of view of the heard-mounted display 425)).
Regarding Claim 7, Lacey in view of Tang and Connellan teaches all of the elements of the claimed invention, as stated above. Furthermore, Lacey teaches:
The information processing apparatus according to claim 1,
Wherein the operating body is the hand of the user (See FIG. 46C, showing examples of when the operating body is the hand of the user), and
wherein in the first acquisition processing, information of the position of the operating body is acquired by detecting the hand of the user from a captured image (FIG. 2B: from camera 124) of the three-dimensional space (See paragraph [0086)).
Regarding Claim 8, Lacey in view of Tang and Connellan teaches all of the elements of the claimed invention, as stated above. Furthermore, Lacey teaches:
The information processing apparatus according to claim 1,
wherein the operating body isa controller (See FIG. 46C, showing examples of when the operating body isa controller), and
wherein in the first acquisition processing, information of a position of the controller acquired by a sensor (FIG. 2B: IMU 102) included in the controller is acquired as information of the position of the operating body (See paragraph [0194]).
Regarding Claim 16, Lacey in view of Tang and Connellan teaches all of the elements of the claimed invention, as stated above. Furthermore, Lacey and Tang teach:
The information processing apparatus according to claim 1, wherein a range that the user's hand can reach is measured and set in advance for each user, and the range is used to determine the predetermined threshold (See Lacey’s paragraph [0428] and FIG. 53; Tang’s [0023, 0055]: the first operation mode is a hand operation mode that set so that the user to perform an operation on the first virtual object by directly touching and selecting the first virtual object with the user's hand within the user’s reach).
Regarding Claim 17, Lacey in view of Connellan teaches all of the elements of the claimed invention, as stated above. Furthermore, Connellan teaches:
The information processing apparatus according to claim 1, wherein determining that the user is not in the predetermined posture in which the operating body is held in front of the user (e.g. 1520b not in the FOV so the mode is set to a lower sensitivity) includes determining, based on posture information of the HMD acquired by a posture sensor (1512 and 1560), that a position of the operating body is outside a frontal region relative to the user (Connellan’s Fig. 15, Pars. 146, 148, 153-154, see also Par. 155). See claim 1 motivation above.
Regarding Claim 18, Lacey in view of Tang and Connellan teaches all of the elements of the claimed invention, as stated above. Furthermore, Lacey teaches:
The information processing apparatus according to claim 1, wherein the operating body is the user's hand (See Lacey’s paragraph [0428]), and wherein in the first acquisition processing, hand articulation points are detected by machine learning (See Lacey’s paragraph [0148, 0149, 0445]) and a direction of the hand is acquired based on information of a plurality of positions on the hand (See Lacey’s paragraph [0428, 0429]).
Regarding Claim 19, Lacey in view of Tang and Connellan teaches all of the elements of the claimed invention, as stated above. Furthermore, Lacey and Tang teach:
The information processing apparatus according to claim 1, wherein the predetermined threshold is set to one half of a width in a horizontal direction of a virtual object at the position of the operating body (See Lacey’s paragraph [0428] and FIG. 53; Tang’s [0023, 0055] and FIG. 6: the first operation mode is a hand operation mode that set so that the user to perform an operation on the first virtual object when it is near and selecting the first virtual object with the user's hand within the user’s reach). Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to try with a reasonable expectation of success to set the predetermined threshold to one half of a width in a horizontal direction of a virtual object at the position of the operating body since it is near the user and thereby reachable by the user’s hand.
Regarding Claim 20, Lacey in view of Tang and Connellan teaches all of the elements of the claimed invention, as stated above. Furthermore, Connellan teaches:
The information processing apparatus according to claim 1, wherein determining that the user is not in the predetermined posture in which the operating body is held in front of the user is performed (e.g. 1520b not in the FOV so the mode is set to a lower sensitivity) based on a spatial relationship between a position of the operating body and posture information of the HMD acquired by a posture sensor (1512 and 1560) (Connellan’s Figs. 14-15, Pars. 123, 148, 153-154, see also Par. 155). See claim 1 motivation above.
Regarding Claim 21, Lacey in view of Tang and Connellan teaches all of the elements of the claimed invention, as stated above. Furthermore, Lacey in view of Tang and Connellan teaches:
The information processing apparatus according to claim 1, wherein, when switching to the first operation mode in the case of (c) (Connellan’s Pars. 148, 153-155 and FIG. 15), displaying the display item is stopped and selection is allowed only when a position of the operating body overlaps a virtual object (See Lacey’s paragraph [0428] and FIG. 53; Tang’s [0052] and FIG. 6);.
Regarding Claim 22, Lacey in view of Connellan teaches all of the elements of the claimed invention, as stated above. Furthermore, Connellan teaches:
The information processing apparatus according to claim 20, wherein the posture sensor comprises an inertial measurement unit (Fig. 15, Par. 146).
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over Lacey in view of Tang and Connellan as applied to claim 1 above, and further in view of Fan et al. (US 20220198756 A1; Previously cited in the PTO-892 dated 09/18/2024), hereinafter Fan.
Claims 1, 3-4, 6-10, 12-13 and 16-22 are rejected under 35 U.S.C. 103 as being unpatentable over Lacey
Regarding Claim 11, Lacey in view of Tang and Connellan does not explicitly teach:
The information processing apparatus according to claim 1, wherein the predetermined threshold is determined based on at least any of (1) a size of a virtual object at the position of the operating body and (2) a distance between the virtual object and another virtual object.
However, in the same field of endeavor, systems for human interaction with an electronic device (Fan, paragraph [0001]), Fan teaches:
A predetermined threshold is determined based on at least any of (1) a size of a virtual object at the position of the operating body and (2) a distance between a virtual object and another virtual object (See FIG. 1A: the virtual objects 22, 22a) (See paragraph [0102]: the predetermined threshold is made smaller in a densely populated XR environment 20).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the information processing apparatus (as taught by Lacey in view of Tang and Connellan ) so the predetermined threshold is determined based on at least any of (1) a size of the first virtual object at the position of the operating body and (2) a distance between the first virtual object and another virtual object (as taught by Fan). Doing so would avoid accidentally selecting an unwanted object, in a densely populated XR environment (See Fan, paragraph [0102]).
Claim 14 is rejected under 35 U.S.C. 103 as being unpatentable over Lacey in view of Tang and Connellan as applied to claim 1 above, and further in view of Smyth. (US 8,824,779).
Regarding Claim 14, Lacey in view of Tang and Connellan does not explicitly teach:
The information processing apparatus according to claim 1, wherein in the second acquisition processing, the line-of-sight position is corrected based on information of a refractive degree of the user's eyes.
Smyth discloses in the second acquisition processing, the line-of-sight position is corrected based on information of a refractive degree of the user's eyes (Col. 43, lines 25-30).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the information processing apparatus (as taught by Lacey in view of Tang and Connellan) to correct the line-of-sight position based on information of a refractive degree of the user's eyes in the second acquisition (as taught by Smyth). Doing so would provide an improved line of sight estimation (See Smyth, Col. 43, lines 25-30).
Claim 15 is rejected under 35 U.S.C. 103 as being unpatentable over Lacey in view of Tang and Connellan as applied to claim 1 above, and further in view of Hiroi et al. (US 2019/0146222 A1).
Regarding Claim 15, Lacey in view of Tang and Connellan does not explicitly teach:
The information processing apparatus according to claim 1, wherein in the second acquisition processing, the line-of-sight position is determined as a three-dimensional position based on a convergence angle derived from line-of-sight directions of left and right eyes and an inter-pupillary distance.
Hiroi discloses in the second acquisition processing, the line-of-sight position is determined as a three-dimensional position based on a convergence angle derived from line-of-sight directions of left and right eyes and an inter-pupillary distance (Fig. 4, Pars. 72-73).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the information processing apparatus (as taught by Lacey in view of Tang and Connellan) to determine the line-of-sight position as a three-dimensional position based on a convergence angle derived from line-of-sight directions of left and right eyes and an inter-pupillary distance (as taught by Hiroi). Doing so would provide an improved detection a set standards may be set (See Hiroi, paragraph [0072]).
Response to Arguments
Applicant’s arguments with respect to claim(s) 1,3-4,6-11 and 14-22have been considered but are moot in view of the new ground(s) of rejection.
Examiner notes that the new claim elements are now addressed by reference Tang, Smyth, or Hiroi as necessitated by amendments. Please see above for full basis of rejection.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JARURAT SUTEERAWONGSA whose telephone number is (571)270-7361. The examiner can normally be reached Monday thru Thursday, 8:30AM to 6:00PM, EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Lun Yi Lao can be reached at 571-272-7671. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JARURAT SUTEERAWONGSA/Examiner, Art Unit 2621
/AMR A AWAD/Supervisory Patent Examiner, Art Unit 2621
Prosecution Insights