DETAILED ACTION
This Office Action is in response to the Preliminary Amendment filed on 02/15/2024.
In the instant application, claims 1-37 are cancelled; Claims 38-53 are newly added; Claims 38, 52 and 53 are amended independent claims; Claims 38-53 have been examined and are pending. This action is made non-final.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Drawings
The drawings submitted on 02/15/2024 are acceptable
Information Disclosure Statement
The information disclosure statements (IDS) submitted on 02/15/2024, 05/30/2024, 09/04/2025, 10/28/2025, 01/02/2026, 02/26/2026 and 07/29/2026 were filed before the mailing date of the first office action on the merits. The submissions are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statements are being considered by the examiner.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned at the time any inventions covered therein were effectively filed absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned at the time a later invention was effectively filed in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 38-44, 46-48 and 51-53 are rejected under 35 U.S.C. 103 as being unpatentable over BEITH et al. (“Beith,” US 2021/0065455), published on March 4, 2021 in view of CHUNG et al. (“Chung,” US 2021/0081104), published on March 18, 2021.
Regarding claim 38, Beith teaches an apparatus (Beith: par. 0047 and Fig. 1; an AR device 102 being worn by the user 100) comprising a display configured to display a virtual environment (Beith: par. 0047 and Fig.1; the AR device 102 has a camera 108 and a display 110 that allows a user to see the real-world scene or environment and also that allows AR content to be displayed thereon) and controller circuitry (Beith: par. 0009; an apparatus for generating virtual content is provided that includes a display, a memory configured to store one or more images, and one or more processors (e.g., implemented in circuitry) and coupled to the memory. Par. 0106; the components of the computing device can be implemented in circuitry) configured to:
detect a location of a hand (Beith: par. 0067 and Fig. 2; the virtual keyboard 228 is displayed as a projection at a position on the display 210 relative to the physical hands 226 of the user. The position at which the virtual keyboard 228 is displayed relative to the hands 226 can be determined by registering the virtual keyboard 228 to one or more landmark points detected on the hands 226. Pars. 0069; before displaying the virtual keyboard 228, the processor of the AR device 202 can register the virtual keyboard 228 to one or more landmark points (also referred to as registration points) detected on the hands 226. Pars. 0071 and 0075; the one or more landmark points on the hands 226 can be used as real-world registration points for positioning the virtual keyboard 228 on the display 210. Par. 0079; the virtual keyboard 228 can be maintained at the position determined using the locations of the landmark points of the hands 226);
provide a virtual keyboard at the location of the hand (Beith: par. 0063; in response to detecting the hands 226, the processor of the AR device 202 can generate the virtual keyboard 228 for display. Par. 0067 and Fig. 2; the user can continue to view the real-world scene through the display 210 while the virtual keyboard 228 is displayed. Par. 0068; one or more AR objects can be displayed on the display 210 of the AR device 202, and the user can input text and/or other characters that can be used to manipulate the one or more objects. The virtual keyboard 228 can receive input from the user in response to the user typing text and/or other characters on the virtual keyboard 228. The input can be processed and displayed in the AR objects 224), [the virtual keyboard being nonlinearly mapped to the hand];
detect a relative movement of at least a portion of the hand (Beith: par. 0082; the virtual keyboard 228 may not move if the fingertips are within a threshold distance of selecting nearby keys. For instance, the virtual keyboard 228 can remain stationary when the fingertips are within the threshold distance of selecting nearby keys);
select a virtual key based on the relative movement (Beith: par. 0088; the processor of the AR device 202 can determine the specific parts of the fingers, and can only accept input when a certain part of a finger is in contact with a part (e.g., a button) of the virtual keyboard 228); and
input a text character associated with the selected key in the virtual environment (Beith: par. 0068 and Fig. 2; one or more AR objects can be displayed on the display 210 of the AR device 202, and the user can input text and/or other characters that can be used to manipulate the one or more objects. The virtual keyboard 228 can receive input from the user in response to the user typing text and/or other characters on the virtual keyboard 228. The input can be processed and displayed in the AR objects 224).
Beith does not appear to teach: the virtual keyboard being nonlinearly mapped to the hand.
However Chung teaches a method that uses a camera to recognize a user’s fingers and hand poses as a way to input text; wherein the virtual keyboard being nonlinearly mapped to the hand (Chung: par. 0058; a plurality of keys may be mapped to each finger of the user. The electronic apparatus 100 may determine a finger that types a key among the plurality of fingers based on a motion of the user’s hand 10 or finger typing the key, and determine one of the plurality of keys mapped to the typing finger based on the motion of the typing finger. Note: multiple keys map to a finger is interpreted as nonlinearly mapped).
Accordingly, it would have been obvious to one of ordinary skill in the art , before the effective filing date of the claimed invention, having the teachings of Chung and Beith in front of them to include the method of determining a typing key based on the motion of the typing finger as disclosed by Chung with the method of displaying a virtual keyboard at the location of user’s hands as taught by Beith to provide an improved interface that accurately and quickly receive inputs through an existing portable device without using a separate physical device or replacing a device (Chung: par. 0008).
Regarding claim 39, Beith and Chung teach the apparatus of claim 38,
Beith and Chung further teach: wherein the controller circuitry is further configured to detect the location of the hand by detecting a location of at least one finger (Chung: par. 0011; identify a key corresponding to the first motion, from among a plurality of keys mapped to the finger, based on a position of the finger by the first motion and a reference point set to the finger, control the display to display information corresponding to the identified key. par. 0127; the plurality of keypoints may correspond to the user’s hand or finger, and may include information about the type of finger and position information. The processor may generate a skeleton representing the shape or motion of a hand or finger by connecting a plurality of keypoints 490 to keypoints of each finger. In this example, the processor may determine the bending degree of the finger through an angle between the keypoints connected to each other in the skeleton, and may determine the position of the finger through the position of the plurality of keypoints);and nonlinearly map the virtual keyboard to the hand by associating a set of virtual keys to each of the at least one finger (Chung: par. 0058; a plurality of keys may be mapped to each finger of the user. The electronic apparatus 100 may determine a finger that types a key among the plurality of fingers based on a motion of the user’s hand 10 or finger typing the key, and determine one of the plurality of keys mapped to the typing finger based on the motion of the typing finger. Note: multiple keys map to a finger is interpreted as nonlinearly mapped. Pars. 0147-0148 and Fig. 6B; a plurality of keys included in the virtual keyboard 650 may be mapped to each of the plurality of fingers).
Regarding claim 40, Beith and Chung teach the apparatus of claim 39,
Beith and Chung further teach: wherein the controller circuitry is further configured to nonlinearly map the virtual keyboard to the hand by aligning the virtual position of one virtual key in the associated set of virtual keys with the location of the associated finger (Chung: pars 0147-0150 and Fig. 6B; the plurality of keys included in the virtual keyboard 650 may be grouped according to a position. In this example, a plurality of keys grouped according to the type of hand and the type of finger may be mapped).
Regarding claim 41, Beith and Chung teach the apparatus of claim 38,
Beith and Chung further teach: wherein the relative movement is relative to a start position (Chung: par. 0217; the processor may identify a relative vector between the keypoints 1024 for the left ring finger where a motion is detected and the reference point 1034L set to the left ring finger where a motion is detected. The processor may calculate an end point of the vector by applying the position of the reference key mapped to the left ring finger where a motion is detected to the vector as a starting point (or initial point), and identify the key located at an end point among the plurality of keys mapped to the finger where a motion is detected as a key corresponding to the motion).
Regarding claim 42, Beith and Chung teach the apparatus of claim 38,
Beith and Chung further teach: wherein the relative movement is relative to a maximum movement (Chung: par. 0127; the processor may determine the bending degree of the finger through an angle between the keypoints connected to each other in the skeleton, and may determine the position of the finger through the position of the plurality of keypoints).
Regarding claim 43, Beith and Chung teach the apparatus of claim 38,
Beith and Chung further teach: wherein the relative movement is relative to a continued movement (Chung: par. 0127; the processor may determine the bending degree of the finger through an angle between the keypoints connected to each other in the skeleton, and may determine the position of the finger through the position of the plurality of keypoints).
Regarding claim 44, Beith and Chung teach the apparatus of claim 38,
Beith and Chung further teach: wherein the relative movement is relative to a feedback (Chung: par. 0156 and Fig. 6C; when the key corresponding to the motion is identified as the W key, the processor may display visual feedback to distinguish the W key 670 from other keys in the virtual keyboard 650).
Regarding claim 46, Beith and Chung teach the apparatus of claim 38,
Beith and Chung further teach: wherein the controller circuitry is further configured to provide tactile feedback (F) when a keypress is detected (Chung: par. 0158; when a key corresponding to the motion is identified, the electronic apparatus may provide the user with a physical feedback according to a key input and may improve a feedback problem that may occur in a virtual interface environment)..
Regarding claim 47, Beith and Chung teach the apparatus of claim 38,
Beith and Chung further teach: wherein the apparatus further comprising a camera at least one sensor (Beith: par. 0047 and Fig.1; the AR device 102 has a camera 108 and a display 110 that allows a user to see the real-world scene or environment and also that allows AR content to be displayed thereon), wherein the controller circuitry of the apparatus is configured to determine the location and to determine the relative movement of the hand by receiving image data from the camera (Beith: par. 0068 and Fig. 2; one or more AR objects can be displayed on the display 210 of the AR device 202, and the user can input text and/or other characters that can be used to manipulate the one or more objects. The virtual keyboard 228 can receive input from the user in response to the user typing text and/or other characters on the virtual keyboard 228. The input can be processed and displayed in the AR objects 224. Par. 0060; the virtual keyboard 228 can be generated and displayed in response to detecting the hands 226 of the user in the field of view of a camera of the AR device 202).
Regarding claim 48, Beith and Chung teach a virtual object presenting system comprising the apparatus of claim 38 (Beith: par. 0047 and Fig. 1; AR device 102) and an accessory device (Beth: par. 0066; a pose of the hands 226 can be determined using one or more sensors located on or near one or more of the hands 226. In one example, at least one sensor can be part of a glove on the right hand of the user and at least one other sensor can be part of a glove on the left hand of a user), the apparatus further comprising a sensor device and the accessory device comprising at least one sensor, wherein the controller circuitry of the apparatus is configured to determine the location and to determine the relative movement of the hand by receiving sensor data from the at least one sensor of the accessory device through the sensor device (Beith: par. 0066; the one or more sensors can detect the pose of the hand and can communicate information for the post to the AR device. The AR device 202 can then displayed relative to the hands of the user as specified by the hand pose information).
Regarding claim 51, Beith and Chung teach the virtual object presenting system of claim 48,
Beith and Chung further teach: wherein the virtual object presenting system comprises two accessory devices (Beth: par. 0066; a pose of the hands 226 can be determined using one or more sensors located on or near one or more of the hands 226. In one example, at least one sensor can be part of a glove on the right hand of the user and at least one other sensor can be part of a glove on the left hand of a user).
Regarding claim 52, this claim is directed toward a method for providing text input in the apparatus, comprising an image presenting device arranged to display a virtual environment (Beith: par. 0047 and Fig. 1; an AR device 102 being worn by the user 100. The AR device 102 has a camera 108 and a display 110 that allows a user to see the real-world scene or environment and also that allows AR content to be displayed thereon), as claimed in claim 38; Claim 51 is similar scope to claim 38 and is therefore rejected under similar rationale.
Regarding claim 53, this claim is directed toward an apparatus comprising a display configured to display a virtual environment and circuitry for providing text input in said virtual environment (Beith: par. 0009; an apparatus for generating virtual content is provided that includes a display, a memory configured to store one or more images, and one or more processors (e.g., implemented in circuitry) and coupled to the memory. Par. 0106; the components of the computing device can be implemented in circuitry); claim 53 is similar scope to claim 38 and is therefore rejected under similar rationale.
Claim 45 is rejected under 35 U.S.C. 103 as being unpatentable over Beith and Chung as applied to claim 38 above and further in view of Zhao et al. (“Zhao” US 10845878, patented on November 24, 2020.
Regarding claim 45, Beith and Chung teach the apparatus of claim 38,
Beith and Chung do not explicitly teach: wherein the controller circuitry is further configured to provide tactile feedback when crossing a delimiter.
However Zhao teaches input device with tactile feedback; wherein the controller circuitry is further configured to provide tactile feedback when crossing a delimiter (Zhao: see col. 7 lines 32-53; the friction-feedback electrodes may particularly indicate a border of the virtual keys 206, or a transition between virtual keys 206).
Accordingly, it would have been obvious to one of ordinary skill in the art , before the effective filing date of the claimed invention, having the teachings of Zhao, Beith and Chung in front of them to include the input device for providing tactile feedback as disclosed by Zhao with the method of displaying a virtual keyboard at the location of user’s hands as taught by Beith to provide variable frictional feedback to tactilely indicate the locations of the user input regions on the input surface and thus adapt to different user needs and preferences (Zhao: see col. 1 lines 21-53).
Claims 49-50 are rejected under 35 U.S.C. 103 as being unpatentable over Beith and Chung as applied to claim 38 above and further in view of Adams et al. (“Adam,” US 9104271), patented on August 11, 2015.
Regarding claim 49, Beith and Chung teach the virtual object presenting system of claim 48,
Beith and Chung do not explicitly teach: wherein the accessory device further comprises one or more actuators for providing tactile feedback, and wherein the controller circuitry of the apparatus is configured to provide said tactile feedback through the at least one of the one or more actuators.
However Adams teaches a method for tracking movement of a gloved hand of a human; wherein the accessory device further comprises one or more actuators for providing tactile feedback, and wherein the controller circuitry of the apparatus is configured to provide said tactile feedback through the at least one of the one or more actuators (Adams: see col. 20 line 54 to col. 21 line 7; movement of a gloved hand of a human can be tracked, the gloved hand wearing a protective glove comprising a sensory output transducer, such as a vibrotactile actuator and/or a tactor. Movement of one or more fingers of the gloved hand can be interpreted, for example, as an actual human finger movement for operating an input device communicatively coupled to information device. In response to interpreting the gloved finger movement, via the sensory output transducer, haptic feedback can be provided to the human, such as vibrotactile feedback that simulates the striking of a key of a keyboard, sliding a mouse, rolling a trackball, scrolling a wheel, and/or clicking a button, etc. The haptic feedback can be provided across a protective layer of the glove to the skin of the human. An output device, such as a display, can render the interpreted finger movement and/or an output corresponding to the interpreted finger movement, such as an entry, positioning, selection, scroll, and/or movement of a key, character, function key, menu button, text, object, cursor, button, and/or window, etc. See col. 3 lines 52-55; tactile feedback can be enabled by a unique concept that can use the pressurized gloves themselves as a transport mechanism for carrying tactile vibrations from motors to human fingers).
Accordingly, it would have been obvious to one of ordinary skill in the art , before the effective filing date of the claimed invention, having the teachings of Adams, Beith and Chung in front of them to include the method of providing tactile feedback via a protective glove as disclosed by Adams with the method of displaying a virtual keyboard at the location of user’s hands as taught by Beith to create an intuitive human-machine interface that can emulate familiar desktop input devices (Adams: see col. 3 lines 44-51).
Regarding claim 50, Beith, Chung and Adams teach the virtual object presenting system of claim 49,
Beith, Chung and Adams further teach: wherein the accessory device is a glove (Beth: par. 0066; a pose of the hands 226 can be determined using one or more sensors located on or near one or more of the hands 226. In one example, at least one sensor can be part of a glove on the right hand of the user and at least one other sensor can be part of a glove on the left hand of a user).
Conclusion
The prior art made of record on form PTO-892 and not relied upon is considered pertinent to applicant's disclosure. Applicant is required under 37 C.F.R. § 1.111(c) to consider these references fully when responding to this action.
It is noted that any citation to specific, pages, columns, lines, or figures in the prior art references and any interpretation of the references should not be considered to be limiting in any way. A reference is relevant for all it contains and may be relied upon for all that it would have reasonably suggested to one having ordinary skill in the art. In re Heck, 699 F.2d 1331, 1332-33,216 USPQ 1038, 1039 (Fed. Cir. 1983) (quoting In re Lemelson, 397 F.2d 1006,1009, 158 USPQ 275,277 (CCPA 1968)).
Inquiry
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Tam T. Tran whose telephone number is (571) 270-5029. The examiner can normally be reached M-F: 7:30 AM - 5:00 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, William L. Bashore can be reached on 571-272-4088. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/TAM T TRAN/Primary Examiner, Art Unit 2174
Prosecution Insights