DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . Claims 1-18 are pending.
Response to Arguments
Applicant's arguments filed 10 August 2026 have been fully considered but they are not persuasive.
Regarding Claims 1, 12, and 13. Applicant states:
Claims 1, 12, and 13 recite, inter alia, detecting, switching a function performed by the display device according to the first detection result (a movement of a hand of a user viewing the image) and the second detection result (a position of the hand in association acquisition of the first detection result).
Yang [0137] discloses, e.g., recognizing that the user’s hand 310 has entered within a distance d1, etc. The Official Action states that Yang does “switch a function [operate in first mode based on hand detected at a distance greater than fig. 3B @d1, perform in a second mode when hand distance is less than fig. 3B @d1] performed by the display device according to the first detection result [hand movement] and the second detection result [hand distance].” However, in Applicant’s most respectful estimation, Yang’s switching of mode turns on distance but does not also depend upon movement in the manner claimed to switch a function performed by the display device.
The Examiner respectfully disagrees with Applicant’s statements.
Claim 1 requires (Claims 12 and 13 are essentially the same):
detect, as a first detection result, a movement of a hand of a user viewing the spatial image displayed on the display device using the display device;
detect, as a second detection result, a position of the hand in association with acquisition of the first detection result; and
switch a function performed by the display device according to the first detection result and the second detection result.
Yang teaches (emphasis added):
[0100] The processor 120 according to an embodiment may be configured to: display the XR content screen on the display 210 in a first mode that displays a hand shape corresponding to a recognized user's hand, when the user's hand is positioned at a distance farther than a first distance spaced apart from the touch sensor 240, through at least one of the first cameras (e.g., first functional camera 220, second functional camera 223); switch to a second mode that displays the user's hand shape in a form of a virtual pointer object, and display the XR content screen on the display 210, based on the user's hand entering within the first distance; and switch to a third mode that displays the virtual area frame to display the XR content screen on the display 210, based on detection of a touch input through the touch sensor 240.
[0121] The first mode (e.g., normal mode/far mode) may mean a mode that displays the user's hand 310 recognized in a camera image as a hand shape, while a user is wearing the HMD device 201. The hand-shaped object 310-1 may be a user's hand that has been actually captured, or may be represented as a virtual hand object, depending on the XR service environment.
[0132] In operation 430, the processor 120 may determine whether the user's hand 310 enters within a first distance d1 from the HMD device 201 or the touch sensor 240.
[0133] For example, the processor 120 may detect an external object (e.g., a user's hand) present in the user's field of view based on images captured through cameras (e.g., first functional camera 220 and/or second functional camera 223 in FIG. 2) facing in a second direction (e.g., second direction (2 in FIG. 2). The processor 120 may determine the movement direction and position of the user's hand 310 through hand tracking.
The flowchart (fig. 4) enters and remains in a loop (steps 430 and 440), tracking the movement direction and position of the user’s hand until the detected hand position is ≤ d1. The claimed function switch (step 450) is triggered by the users hand moving from a position ˃ d1 to a position ≤ d1.
The input device taught by Yang can start the process of figure 4 under two conditions:
The first detected hand position p1 is ≤ d1, the first instance of fig. 4 step 430 is YES and the function switch (step 450) is executed based on:
(a) a first detection result - recognizing and tracking the user’s hand and
(b) a second detection result - user’s hand position ≤ d1.
The first detected hand position p1 is ˃ d1, the first instance of fig. 4 step 430 is NO and a function switch does not occur. The processor continues to tracking user’s hand (determining hand movement direction and position)
Time Hand Position Function Switch
1 p1 ˃ d1 No
2 p2 ˃ d1 No
3 p3 ˃ d1 No
4 p4 ≤ d1 YES
The function switch [fig. 4 @450] is executed based on:
(a) a first detection result p3 ˃d1 (movement of the user’s hand until the second detection result)
(b) a second detection result p4 ≤ d1 (distance)
The detection of a plurality of positions over time is a detection of movement. Without detecting hand movement from the latest hand position ˃d1 to the current hand position ≤ d1, the function switch will not occur.
Yang teaches:
detect, as a first detection result, a movement of a hand of a user viewing the spatial image displayed on the display device using the display device;
detect, as a second detection result, a position of the hand in association with acquisition of the first detection result; and
switch a function performed by the display device according to the first detection result and the second detection result.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-6, 9, and 11-14 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Yang (US 2026/0016951).
Regarding Claims 1, 12, and 13 (Original), Yang teaches an information processing apparatus, a method of operating an information processing apparatus, and a non-transitory computer-readable medium comprising computer-executable instructions stored thereon that, when executed by a computer, cause the computer to perform a method of controlling an information processing apparatus configured to:
be communicatively connected to a display device [¶0117, “The HMD device 201 may output the XR content screen 320 through a display (e.g., display module 160 in FIG. 1, display 210 in FIG. 2)”] that displays a spatial image including at least one of a real-space image and a virtual-space image [¶0118, “The XR content screen 320 may include at least one virtual object 321 in a background space 325 (e.g., real space or virtual space). For example, the HMD device 201 may configure the background space 325 as an area determined to be a field of view (FoV) of a user, based on a viewpoint of the user, and may generate the XR content screen 320 by rendering the virtual object 321 included in the area determined to be the field of view (FoV) of the user. The background space 325 may be a real image captured through a camera, depending on an XR service environment, or may be an image of a virtual environment”] and configured to:
process information communicated with the display device, the information processing apparatus comprising one or more processors [fig. 1 @120] configured to:
detect, as a first detection result, a movement of a hand [¶0133, “the processor 120 may detect an external object (e.g., a user's hand) present in the user's field of view based on images captured through cameras (e.g., first functional camera 220 and/or second functional camera 223 in FIG. 2) facing in a second direction (e.g., second direction (2 in FIG. 2). The processor 120 may determine the movement direction and position of the user's hand 310 through hand tracking”, ¶0136, “the HMD device 201 may determine that an interaction is performed on the virtual object 321, based on the user's hand 310 overlapping the position where the virtual object 321 is displayed and a gesture for grabbing (or grab gesture) the virtual object 321 being recognized”] of a user [fig. 3A @201] viewing the spatial image [fig. 3A @320] displayed on the display device using the display device;
detect, as a second detection result, a position of the hand in association with acquisition of the first detection result [¶0119, “The screen of FIG. 3A illustrates the XR content screen 320 in a first mode that is displayed to a user when a user's hand 310 is positioned beyond a first distance spaced apart from a touch sensor (e.g., touch sensor 240 in FIG. 2) (or the HMD device)”]; and
switch a function [operate in first mode based on hand detected at a distance greater than fig. 3B @d1, perform in a second mode when hand distance is less than fig. 3B @d1] performed by the display device according to the first detection result [hand movement] and the second detection result [hand distance].
Regarding Claim 2 (Original), Yang teaches the information processing apparatus according to Claim 1, wherein
a hand gesture is detected as the first detection result [¶0136].
Regarding Claim 3 (Original), Yang teaches the information processing apparatus according to Claim 1, wherein
a distance between the hand and the display device is detected as the second detection result [¶0119].
Regarding Claim 4 (Currently Amended), Yang teaches the information processing apparatus according to Claim 1, wherein
the function performed by the display device includes a plurality of functions [¶0011, “display the XR content screen on the display in a first mode in which a hand-shaped object corresponding to the hand of the user is displayed in the XR content screen; based on identifying that the hand of the user is positioned within the first distance, display the XR content screen on the display in a second mode in which the hand-shaped object is changed to the virtual pointer object; and based on detection of a touch input via the touch sensor, displaying the XR content screen on the display in a third mode in which the virtual area frame is displayed in the XR content screen”],
wherein a hand gesture is detected as the first detection result [¶0136],
wherein a distance between the hand and the display device is detected as the second detection result [¶0119],
wherein when the hand gesture is detected and the detected distance is less than a threshold [¶0122, “The screen of FIG. 3B illustrates the XR content screen 320 in a second mode that is displayed to a user when the user's hand 310 enters within a first distance d1 spaced apart from a touch sensor“] the function performed by the display device is switched to a first function [second mode], which is one of the plurality of functions, and
wherein when the hand gesture is detected and the detected distance is equal to or greater than the threshold [¶0119, “The screen of FIG. 3A illustrates the XR content screen 320 in a first mode that is displayed to a user when a user's hand 310 is positioned beyond a first distance spaced apart from a touch sensor (e.g., touch sensor 240 in FIG. 2) (or the HMD device)”], the function performed by the display device is switched to a second function [first mode], which is one of the plurality of functions and is different from the first function.
Regarding Claim 5 (Original), Yang teaches the information processing apparatus according to Claim 4, wherein
the one or more processors are further configured to change the threshold [from d1 to physical touch, ¶0132, “In operation 430, the processor 120 may determine whether the user's hand 310 enters within a first distance d1 from the HMD device 201 or the touch sensor 240, ¶0141, “In operation 460, the processor 120 may determine whether a touch input directly contacting the touch sensor 240 is detected”].
Regarding Claim 6 (Original), Yang teaches the information processing apparatus according to Claim 4, wherein the one or more processors are further configured to:
cause the display device to display an indication [fig. 3B @330] that switching to the first function [second mode] is possible when the function performed by the display device becomes [when distance criteria met] switchable to the first function [second mode], and
cause the display device to display an indication [fig. 3A @310-1] that switching to the second function [first mode] is possible when the function performed by the display device becomes switchable [distance criteria met] to the second function [first mode].
Regarding Claim 9 (Currently Amended), Yang teaches the information processing apparatus according to Claim 1, wherein
the display device includes an imager [fig. 2 @220] configured to capture an image of the hand [¶0081, “The first functional camera 220 may support at least one of head tracking, hand detection, hand tracking, or spatial recognition”], and wherein the one or more processors are further configured to:
(1) detect, as the first detection result, the movement of the hand based on the image of the hand [¶0136]; and
(2) detect, as the second detection result, the position of the hand based on the image of the hand [¶0119].
Regarding Claim 11 (Currently Amended), Yang teaches the information processing apparatus according to Claim 1, wherein
the display device is (1) a head-mounted display configured to be worn on the user's head [¶0066, “FIG. 2 schematically illustrates an external appearance of a head-mounted display (HMD) device according to an embodiment. 2001 in FIG. 2 may be an external appearance of an HMD device 201 viewed from a first direction”] or (2) a handheld display configured to be held by at least one of the user's left and right hands [alternate limitation not addressed].
Regarding Claim 14 (New), Yang teaches the information processing apparatus according to Claim 4, wherein
the first function [fig. 3A @310, ¶0100, “a first mode that displays a hand shape corresponding to a recognized user's hand, when the user's hand is positioned at a distance farther than a first distance spaced apart from the touch sensor 240, through at least one of the first cameras (e.g., first functional camera 220, second functional camera 223)”] and
the second function [fig. 3B @310, ¶0100, “a second mode that displays the user's hand shape in a form of a virtual pointer object, and display the XR content screen on the display 210, based on the user's hand entering within the first distance”] are assigned to a same type of the hand gesture [hand movement from first position to second position, illustrated by change from fig. 3A @310 to fig. 3B @310 or from fig. 3B @310 to fig. 3A @310], and
wherein the one or more processors are configured to select one of the first function [first mode if second position d ˃ d1] and the second function [second mode if second position d ≤ d1] based on the detected distance in response to detection of the same type of the hand gesture.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 7-8, and 10 are rejected under 35 U.S.C. 103 as being unpatentable over Yang in view of Whitmire (US 2024/0019938). All reference is to Yang unless otherwise indicated.
Regarding Claim 7 (Original), Yang teaches the information processing apparatus according to Claim 4
Yang does not teach display a boundary line that the hand crosses when the function performed by the display device is switched between the first function and the second function
Whitmire teaches display a boundary line [fig. 1A @106] that a hand crosses when a function performed by the display device is switched between a first function [¶0070, “The hold gesture can also be referred to as a stateful gesture above the virtual object 118, beyond the surface threshold distance 106 but within the in-air threshold distance 108”] and a second function [¶0069, “The surface gesture 122 is performed within the surface threshold distance 106 of the physical surface 110”]
Before the application was filed it would have been obvious to one of ordinary skill in the art to incorporate the concept of displaying a boundary line delineating the transition points where the apparatus changes between performing a first function to performing a second function, as taught by Whitmire into the apparatus, taught by Yang, in order to enable a user wearing an HMD to control a device in a desired mode by visual reference to their detected hand position.
Regarding Claim 8 (Original), Yang in view of Whitmire teaches the information processing apparatus according to Claim 7, wherein the one or more processors are further configured to
cause the display device to display the boundary line [Whitmire: fig. 1C @106] when the hand is moving [Whitmire: fig. 1C @122 illustrates movement gesture].
Regarding Claim 10 (Currently Amended), Yang teaches the information processing apparatus according to Claim 1
Yang does not teach a communication device that is communicatively connected to the information processing apparatus is worn on the hand, and wherein the one or more processors are further configured to: (1) detect, as the first detection result, the movement of the hand based on communication information from the communication device; (2) detect, as the second detection result, the position of the hand based on communication information from the communication device
Whitmire teaches a communication device that is communicatively connected to the information [Whitmire: ¶0021, “ … the wrist-wearable device is communicatively coupled with a head-wearable device”] processing apparatus is worn on the hand [the wrist is directly connected to the hand and movement of the wrist is equivalent and relatable to movement of the hand, fig. 1D @102], and wherein the one or more processors are further configured to:
(1) detect, as the first detection result, the movement [fig. 1D @123] of the hand [¶0070] based on communication information from the communication device [fig. 1D @102];
(2) detect, as the second detection result, the position of the hand [fig. 1D illustrates device 102 between 108 and 106] based on communication information from the communication device [fig. 1D @102, ¶0070, “FIG. 1D shows the user 101 performing an in-air gesture 123. Specifically, FIG. 1D shows a hold gesture where the user's hand stays in one position (e.g., a constant wrist position) for a predefined hold period above. For example, the predefined hold period can be 1 second to 1.5 seconds … The hold gesture can also be referred to as a stateful gesture above the virtual object 118, beyond the surface threshold distance 106 but within the in-air threshold distance 108”]
Before the application was filed it would have been obvious to one of ordinary skill in the art to incorporate the concept of a wearable device configured to communicate movement and position information to a processor, as taught by Whitmire, into the apparatus, taught by Yang, in order to utilize the sensors contained in the wearable device to track the movement and position of a user’s wrist without requiring a dedicated camera and the associated image processing required to track a user’s hand.
Claim 18 is rejected under 35 U.S.C. 103 as being unpatentable over Yang in view of Aratani (US 2012/0002841). All reference is to Yang unless otherwise indicated.
Regarding Claim 18 (New), Yang teaches the information processing apparatus according to Claim 11, wherein
the one or more processors are configured to detect a movement [¶0133, “… The processor 120 may determine the movement direction and position of the user's hand 310 through hand tracking”] and
a position of the user's detected hand [¶0134, “In operation 440, when the user's hand 310 is positioned at a distance farther than the first distance d1 from the touch sensor 240 or the HMD device 201”] as the first detection result and the second detection result, respectively
Yang does not teach the display device is the handheld display held by one of the user's left and right hands, and wherein the user’s detected hand is the other hand
Aratani teaches a handheld display [¶0081, “… the three-dimensional position measurement apparatus 200 according to the present exemplary embodiment is attached to … a handheld display to apply it to the MR technology”] held by one of the user's left and right hands [construed as at least one hand is required to hold the handheld device], and wherein
the other hand [construed as the hand, of the right and left hands, not holding the handheld display] is the user’s detected hand [fig. 2 @150]
Before the application was filed it would have been obvious to one of ordinary skill in the art to incorporate the handheld three dimensional tracking apparatus, taught by Aratani, into information processing apparatus taught by Yang, in order to apply mixed reality technology to a portable device that does not require the user to wear it.
Allowable Subject Matter
Claims 15-17 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a). A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the Examiner should be directed to Douglas Wilson whose telephone number is (571)272-5640. The Examiner can normally be reached 1000-1800 EST. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the Examiner by telephone are unsuccessful, the Examiner’s supervisor, Patrick Edouard can be reached at 571-272-7603. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Douglas Wilson/Primary Examiner, Art Unit 2622
