Detailed Action
Notice of Pre-AIA or AIA Status
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
2. The amendment filed 4/14/26 has been entered. Claims 1-3, 5-13, 15-22, 24-31 are pending.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
4. The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”), such as in claims 20-22, 24, and 26-28, are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
6. Claims 1-3, 5-13, 15-22, 24-31 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. Independent claims 1, 11, 20, and 29 recite “selecting a pair of keys on a physical keyboard based on the tracking information and the predicted target user interface element; and adjusting a speed or acceleration of the cursor based on a distance from the cursor to the predicted target user interface element and the tracking information so that two fingers of the hand of the user will be on or over the pair of keys on the physical keyboard when the cursor is on the predicted target user interface element” it does not make sense that the adjustment of the cursor speed or acceleration will affect the placement of the hand over the pair of keys. Rather, the causality in this recitation is reversed. It should recite that the speed/acceleration of the cursor is adjusted so that the cursor is on the predicted target user interface element when two fingers of the hand of the user will be on or over the pair of keys on the physical keyboard. This is how the claims will be interpreted for purposes of examination, but appropriate correction is required. The dependent claims do not remedy the issue and are rejected as well. Likewise, although selecting a pair of keys on a physical keyboard would be based on the tracking information, nevertheless to say it is also based on the predicted target user interface element is circuitous and confusing, because the user interface element prediction itself is tied to the actual hand/finger movement and physical key prediction, which is based on the tracking information. For purposes of examination, this feature of physical key selection will be interpreted as ultimately being based on the tracking information and physical movement of the hand/fingers with respect to the physical keys on the keyboard, as is then virtually represented by the cursor and virtual keyboard. Appropriate correction is still required.
Claim Rejections - 35 USC § 103
7. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
8. Claim(s) 1-2, 5-9, 11-12, 15-18, 20-21, 24-27, and 29 is/are rejected under 35 U.S.C. 103 as being unpatentable over Grossinger et al “Grossinger” (KR 101711619 B1) and Foresti (US 20180173417 A1) and Cohen et al “Cohen” (KR 2008/0106265 A).
(Please see the attached copies of Grossinger and Cohen that number paragraphs in the same manner as that used in this Action).
9. Regarding claim 1, Grossinger shows a method for receiving user inputs to a computing device, the method comprising: receiving tracking information corresponding to movement of a hand of a user while operating in a virtual mouse operating mode (para 147, 199, 227 show operating in a virtual mouse mode where the image of the mouse is displayed and acted upon. Para 150, 157 show that while in this mode the hand tracker mechanism may be used. Figure 6, para 187, 193, 252, 332 show the hand tracker obtaining information tracking movement of a hand of a user); generating one or more instructions to move a cursor of the virtual mouse in a graphical user interface (GUI) based on the tracking information (Figure 6, para 192-193, 196-199 show moving the cursor corresponding to a virtual mouse in a graphical user interface (GUI) based on the tracking information). Grossinger does not explicitly show predicting a target user interface element for the cursor in the GUI; and adjusting a speed or acceleration of the cursor based on a distance to the predicted target user interface element so that two fingers of the hand of the user will be on or over a pair of keys on the physical keyboard, selected based on the tracking information and predicted target user interface element, when the cursor is on the predicted target user interface element. Please see also the 112 rejection and interpretation of this feature. Foresti however does show predicting a target user interface element for the cursor in the GUI (para 42, 46, 91 show predicting which virtual target key for the cursor to be placed for selection); and adjusting a speed or acceleration of the cursor based on a distance from the cursor to the predicted target user interface element and tracking information (Figure 4, para 42, 46, 91 show adjusting the speed of the cursor based on a distance to the predicted virtual target key, para 25, 27, 72, 91 show the tracking information also used to adjust the cursor speed). It would have been obvious to a person with ordinary skill in the art before the effective filing date of the claimed invention to predict a target key for the cursor to be placed for selection and to adjust the speed of the cursor based on a distance to the predicted target key as is done in Foresti, in the virtual mouse with hand tracking method of Grossinger, because it would provide an efficient way to track the user’s hand/fingers and move the cursor corresponding to the virtual mouse accordingly. Doing so would minimize potential lag and misalignment of the tracking by matching virtual movement to physical movement. Nevertheless, neither Grossinger nor Foresti show this explicitly is used so that two fingers per se of the hand of the user will be on or over a pair of keys on a physical keyboard, selected based on the tracking information and predicted target user interface element, when the cursor is on the target user interface element (aka the virtual target key), but Grossinger para 348 does show the virtual keyboard, as does Foresti para 23, 25, 27 for example. Grossinger para 291 also shows specifically tracking the movement of a user’s finger. Furthermore, Cohen does show a virtual keyboard that is aligned to and based on a physical keyboard so that two fingers of a hand of the user will be on a pair of keys on the physical keyboard when the cursor is on a target user interface element (para 97, 121 show the cursor may be a virtual hand being moved onto graphical elements. Figure 2B, para 115, 118 show fingers of a real hand such that when a finger moves from one physical key to another on an actual keyboard, the corresponding virtual hand/cursor moves to a relative virtual key on the virtual keyboard. Para 129 shows the position and movement of the virtual hand is adjusted to make sure a corresponding virtual finger is on the target virtual character key. Para 121-122 show that two physical keys may be covered or pressed to move the virtual hand/cursor onto a target virtual character key of a modified virtual keyboard). It would have been obvious to a person with ordinary skill in the art before the effective filing date of the claimed invention to have two fingers of the hand of the user on two keys on a physical keyboard when the virtual cursor element is on the target user interface element as is done in Cohen, in the virtual mouse/cursor with hand tracking method of Grossinger as modified by Foresti, because it would provide an efficient way to track the user’s hand/fingers and move the cursor (aka now a virtual hand) accordingly along a virtual keyboard. Doing so would minimize misalignment of the tracking by matching virtual movement and selection on a virtual keyboard to physical movement and selection combinations along a physical keyboard.
10. Claim 11 shows the same features as claim 1 and is rejected for the same reasons. Additionally, Grossinger para 150, 178-179, 244 show the camera sensors to track movements of a hand of a user. Claim 1 explains the motivation to have the physical keyboard of Cohen. Grossinger para 186, 322 show the processor coupled to the camera sensor and configured with processor executable instructions to perform the method steps. Cohen para 101 also shows computer processing devices configured with processor executable instructions to perform the method steps. The motivation to combine Grossinger and Cohen is explained for claim 1. Furthermore, it would have been obvious to a person with ordinary skill in the art before the effective filing date of the claimed invention to have one processor (for example by combining the processors of Grossinger and Cohen) so that the same processor would be coupled to the camera sensor and physical keyboard, because it would provide efficient control of the system and minimize processing time.
11. Claim 20 shows the same features as claim 1 and is rejected for the same reasons. Additionally, Grossinger para 186, 322 show the processor means for performing the method steps.
12. Claim 29 shows the same features as claim 1 and is rejected for the same reasons. Additionally, Grossinger para 213-214 show the hardware memory non-transitory processor-readable medium storing processor executable instructions and Grossinger para 186, 322 show the processor of a computing device receiving instructions for performing the method steps.
13. Regarding claim 2, in addition to that mentioned for claim 1, Grossinger shows receiving an indication of a user gesture corresponding to a request for a virtual mouse (Grossinger para 180-181 and 209-210 shows the user gesture indicating the request for the virtual mouse to track and respond to hand movements in a particular manner); and switching from a normal operating mode to the virtual mouse operating mode in response to receiving the indication of the user gesture (Grossinger para 180-181 and 209-210 show switching then to a mode for the virtual mouse to track and respond to hand movements in a particular manner). Grossinger and Foresti do not explicitly show the virtual mouse operating mode is configured to translate one or more key presses of a physical keyboard into mouse clicks of the virtual mouse. Cohen however does show receiving an indication of a user gesture corresponding to a request for a virtual mouse (Cohen para 117-119, 127 show the user makes a gesture to enter a virtual mouse mode); and switching from a normal operating mode to the virtual mouse operating mode in response to receiving the indication of the user gesture (Cohen para 117-119 show the mode switching to a particular virtual mouse mode in response to the gesture), the virtual mouse operating mode configured to translate one or more key presses of a physical keyboard into mouse clicks of the virtual mouse (Cohen para 120-122 show in the mode resulting from the detected gesture, one or a combination of key presses are translated into mouse clicks on modified virtual keys). It would have been obvious to a person with ordinary skill in the art before the effective filing date of the claimed invention to translate one or more key presses of a physical keyboard into mouse clicks of the virtual mouse, as is done in Cohen, in the virtual mouse/cursor with hand tracking method of Grossinger as modified by Foresti, because it would provide an efficient way to use a hand tracker/virtual mouse to interact on a GUI and in particular select a particular key along a virtual keyboard. Doing so would allow a selection of a modified virtual key or other element that may not correspond directly to only a physical location of the user’s hand/finger.
14. Claims 12 and 21 each shows the same features as claim 2 and each is rejected for the same reasons.
15. Regarding claim 5, in addition to that mentioned for claim 1, Grossinger para 147, 199, 227 show operating in the virtual mouse operating mode as explained in claim 1. Cohen shows the one or more key presses of the physical keyboard is translated from a hardware key code of the physical keyboard into the mouse clicks at the computing device (para 129 shows the user pressing a key on the physical keyboard, and the corresponding generated signal is translated by the computer to be the virtual mouse clicking via a virtual finger/hand on a virtual key). It would have been obvious to a person with ordinary skill in the art before the effective filing date of the claimed invention to have this in the virtual mouse/cursor with hand tracking method of Grossinger as modified by Foresti, because it would provide an efficient way to track the user’s hand/fingers and act on virtual elements using the cursor (aka now a virtual hand) accordingly along a virtual keyboard.
16. Regarding claim 6, Grossinger shows receiving GUI information including one or more positions of clickable elements displayed in the GUI (para 196-197, 199-200 show receiving GUI information regarding positions of clickable buttons, menu options, and keys on a virtual keyboard. Grossinger does not show predicting the target user interface element further comprises calculating a vector for the cursor based on the tracking information; and identifying the predicted target user interface element based on the vector and the one or more positions of the clickable elements. Foresti however does show predicting the target user interface element further comprises calculating a vector for the cursor based on the tracking information (para 25 shows tracking hand information para 46, 91 show predicting a target user interface element such as a virtual key among clickable virtual elements. Para 48-50, 58 show the predicting of the virtual key includes calculating a vector for the cursor based on the tracking information); and identifying the predicted target user interface element based on the vector and the one or more positions of the clickable elements (para 42, 46, 48-50 show the predicted virtual key is identified using the vector information and position of the virtual key). It would have been obvious to a person with ordinary skill in the art before the effective filing date of the claimed invention to have this in the virtual mouse/cursor with hand tracking method of Grossinger, because it would provide an efficient way to track the user’s hand/fingers and move the cursor corresponding to the virtual mouse accordingly. Doing so would minimize potential lag and misalignment of the tracking by statistically matching virtual movement to physical movement.
17. Claims 15 and 24 each shows the same features as claim 6 and each is rejected for the same reasons.
18. Regarding claim 7, in addition to that mentioned for claim 1, Cohen shows the movement of the cursor is confined to movement of the hand of the user over the physical keyboard (Figures 1A-B, 2B show the input movement space for the hand is the region over the physical keyboard, and para 115, 118 show when a user’s hand traverses throughout the actual keyboard, the corresponding virtual hand/cursor moves throughout the displayed image. Para 120 shows how the modified virtual keyboard corresponds to the movement of the user’s hand over the physical keyboard). It would have been obvious to a person with ordinary skill in the art before the effective filing date of the claimed invention to confine the movement of the cursor to movement of the user’s hand over the physical keyboard as is done in Cohen, in the virtual mouse/cursor with hand tracking method of Grossinger especially as modified by Foresti, because it would provide an efficient way to track the user’s hand/fingers and move the cursor (aka now a virtual hand) accordingly along a virtual keyboard. Doing so would minimize misalignment of the tracking by matching virtual movement along a virtual keyboard to direct physical movement along a physical keyboard.
19. Claims 16 and 25 each shows the same features as claim 7 and each is rejected for the same reasons.
20. Regarding claim 8, in addition to that mentioned for claim 1, Grossinger para 111, 239, 242-243 show mapping positions in a first area of the hand to a second area in the GUI and as noted for claim 1, Grossinger Figure 6, para 187, 193, 252, 332 show the hand tracker obtaining information tracking movement of a hand of a user and Figure 6, para 192-193, 196-199 show moving the cursor corresponding to a virtual mouse in a graphical user interface (GUI) based on the tracking information. Grossinger and Foresti do not show the first area is of a physical keyboard. Cohen however para 95, 105, 118 show mapping positions in a first area of the user’s hand to a second virtual keyboard area and para 106-107 specifically show this first area is of a physical keyboard. Also as noted for claim 1, Cohen Figures 1A-B, 2B show the input movement space for the hand is the region over the physical keyboard, and para 115, 118 show when a user’s hand traverses throughout the actual keyboard, the corresponding virtual hand/cursor moves throughout the displayed image. It would have been obvious to a person with ordinary skill in the art before the effective filing date of the claimed invention to map positions in a first area of the physical keyboard as is done in Cohen, in the virtual mouse/cursor with hand tracking method of Grossinger especially as modified by Foresti, because it would provide an efficient way to track the user’s hand/fingers and move the cursor accordingly along a virtual keyboard. Doing so would minimize misalignment of the tracking by matching virtual movement along a virtual keyboard to direct physical movement along a physical keyboard.
21. Claims 17 and 26 each shows the same features as claim 8 and each is rejected for the same reasons.
22. Regarding claim 9, note the alternative recitation. Grossinger Figure 11, para 150, 244 show the camera to track movements of a hand of a user in a predefined physical space such as the region above the table. Also, Cohen Figures 1A-B, 2B show the input movement space for the hand is the region over the physical keyboard, and para 115, 118 show when a user’s hand traverses throughout the actual keyboard, the corresponding virtual hand/cursor moves throughout the displayed image. Para 120 shows how the modified virtual keyboard corresponds to the movement of the user’s hand over the physical keyboard). It would have been obvious to a person with ordinary skill in the art before the effective filing date of the claimed invention to track movement of the user’s hand over the predefined physical space such as a physical keyboard as is done in Cohen, in the virtual mouse/cursor with hand tracking method of Grossinger especially as modified by Foresti, because it would provide an efficient way to track the user’s hand/fingers and move the cursor (aka now a virtual hand) accordingly along a virtual keyboard. Doing so would minimize misalignment of the tracking by matching virtual movement along a virtual keyboard to bounded physical movement along a physical region.
23. Claims 18 and 27 each shows the same features as claim 9 and each is rejected for the same reasons.
24. Claim 3, 13, and 22 is/are rejected under 35 U.S.C. 103 as being unpatentable over Grossinger and Foresti and Cohen and Kumar et al “Kumar” (US 2015/0277580 A1).
25. Regarding claim 3, in addition to that mentioned for claim 2, Cohen para 106-107 also show mapping a physical keyboard area to virtual keyboard, para 120-122 show identifying the one or more key presses to translate, and para 125-126 show the key presses are identified by tracking information detected by the camera. Grossinger and Foresti and Cohen do not show switching to the special virtual mouse operating mode disables at least the two keys of the physical keyboard for character input. Kumar para 16, 18, 20, 30 however do show disabling at least two keys of the physical keyboard for character input when switching operating modes such as to tablet mode. Para 18 and 30 specifically shows this is done to avoid an inadvertent key or function selection due to remapping the keyboard keys. It would have been obvious to a person with ordinary skill in the art before the effective filing date of the claimed invention to disable at least two keys of the physical keyboard for character input when switching operating modes as is done in Kumar, in the virtual mouse/cursor with hand tracking method of Grossinger especially as modified by Foresti and Cohen, because it would provide an efficient way to avoid an inadvertent key or function selection due to a remapping of the keyboard such as what occurs in Cohen para 124.
26. Claims 13 and 22 each shows the same features as claim 3 and each is rejected for the same reasons.
27. Claim(s) 30 is/are rejected under 35 U.S.C. 103 as being unpatentable over Grossinger and Foresti and Cohen and Kotta et al “Kotta” (US 2019/0041995 A1).
28. Regarding claim 30, in addition to that mentioned for claim 1, Grossinger and Foresti and Cohen do not explicitly show disabling one or more keys located in a vicinity of the (selected) pair of keys. Kotta does however show disabling one or more keys located in a vicinity of a selected key (para 25 shows disabling the keys not predicted to be selected, in the vicinity of the selected/predicted key). It would have been obvious to a person with ordinary skill in the art before the effective filing date of the claimed invention to disable one or more keys located in the vicinity of the selected pair of keys, in the virtual mouse/cursor with hand tracking method of Grossinger especially as modified by Foresti and Cohen, because it would provide an efficient way to avoid an inadvertent key selection.
29. Claim(s) 31 is/are rejected under 35 U.S.C. 103 as being unpatentable over Grossinger and Foresti and Cohen and Franz et al “Franz” (US 6040821 A).
(Please also see the attached copy of Franz that numbers paragraphs in the same manner as that used in this Action).
30. Regarding claim 31, in addition to that mentioned for claim 1, Cohen para 121 shows predicting a landing position of the hand. Grossinger and Foresti and Cohen do not explicitly show determining that the landing position is outside of an area of the keyboard and further adjusting at least one of the speed and the acceleration of the cursor. Franz however does show determining the landing position is outside of an area of the keyboard and further adjusting at least one of the speed and the acceleration of the cursor (para 15, 121-124, 134 show that when the landing position is away from the keyboard such as due to pointing, the system adjusts [which includes stopping] the speed of the cursor). It would have been obvious to a person with ordinary skill in the art before the effective filing date of the claimed invention to determine the landing position of a hand/finger is outside of an area of the keyboard and further adjusting at least one of the speed and the acceleration of the cursor, in the virtual mouse/cursor with hand tracking method of Grossinger especially as modified by Foresti and Cohen, because it would provide an efficient way to avoid selecting an unintended key or to attempt selection outside of the keyboard (Franz para 8, 123).
Allowable Subject Matter
31. Claims 10, 19, and 28 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims, and if the 112 issues were remedied.
The features of claim 10 including training a virtual mouse process to obtain the tracking information of a particular user in a training session by: generating one or more icon shapes in the GUI for the particular user to click; receiving tracking information corresponding to movement of a hand of the particular user; generating one or more instructions to move a cursor of the virtual mouse in a GUI based on the tracking information; predicting one of the icon shapes that the particular user is likely to click on in the GUI; adjusting the speed or acceleration of the cursor based on a distance to the predicted one of the icon shapes so that two fingers of the hand of the user will be on or over two selected keys on the physical keyboard when the cursor is on the predicted target user interface element; receiving key press information resulting from the particular user pressing two or more keys; and adjusting a parameter used in adjusting the speed or acceleration of the cursor in response to the received key press information indicating that the particular user pressed one or more keys different from the two selected keys on the physical keyboard, combined with the features of the independent claim 1, together distinguish over the prior art of record. As noted prior art such as Grossinger shows virtual mouse and hand tracking processes, Foresti adjusts a speed of the cursor based on a distance to the predicted target element, and Cohen shows virtual keyboard selection corresponding to hand movements and selection over a physical keyboard. Additionally, prior art like Miller and Berliner how virtual keyboard systems that even train a virtual mouse and cursor process to obtain tracking information and predict target icons, and Bradski and Miller even take a target shape into account when targeting and predicting elements. Nevertheless, these and the other prior art alone and/or in combination do not show all of the features as described and recited in claim 10. Claims 19 and 28 each shows the same features as claim 10 and the same rationale applies to each of them.
Response to Arguments
32. Applicant's arguments filed 4/14/26 have been fully considered but they are not persuasive. In view of the amendment, the previous 112 rejection has been removed. However, please note the newly presented 112 rejection as explained in the Action. Regarding the 103 rejection, Applicant describes the prior art and then argues that Grossinger, Foresti, and Cohen do not show “selecting a pair of keys on a physical keyboard based on the tracking information and the predicted target user interface element,” as recited in the independent claims. Please note again the 112 rejection and interpretation of this feature. As explained for claim 1, Grossinger does show the tracking information and Foresti shows predicting a target user interface element for the cursor. Note that Cohen Figure 2B, para 115, 118 show fingers of a real hand such that when a finger moves from one physical key to another on an actual keyboard, the corresponding virtual hand/cursor moves to a relative virtual key on the virtual keyboard. Para 129 shows the position and movement of the virtual hand is adjusted to make sure a corresponding virtual finger is on the target virtual character key. Para 121-122 show that two physical keys may be covered or pressed to move the virtual hand/cursor onto a target virtual character key of a modified virtual keyboard, and note para 121-122 show that these two physical keys may be predicted. The claim feature “selecting a pair of keys on a physical keyboard based on the tracking information and the predicted target user interface element” really means selecting the pair of keys as the predicted pair of keys that the user’s fingers will go over. Thus Cohen shows “selecting” these physical keys (as being predicted) as well. The selection of predicted keys as recited in Applicant’s claim is based on tracking information and the predicted target user interface element, but this predicted target user interface element is itself based on the movement of the cursor which is based on movement of the hand and its distance to the actual physical keys on the keyboard. Herein lies the connection between the 112 and 103 rejections: although the virtual representation (aka the cursor and user interface element) may be used for the prediction as recited in the claims, it nevertheless is tied to the actual hand/finger movement and physical key prediction, which is based on the tracking information. Applicant’s recitation and argument assumes that the physical selection of the keys somehow is based on some independent predicted target user interface element but that is not the case. Cohen shows the predicting of the pair of physical keys and selecting these predicted keys. The Action explains thus how the combination of the prior art, including the tracking features of Grossinger and prediction features of Foresti, combined with Cohen, indeed show the virtual mouse/cursor with hand tracking method as recited in the independent claims.
Conclusion
33. Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
34. The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
a) Theimer (CA 2732550) interprets a user’s hand or finger gestures hovering above a screen into cursor position on a GUI.
b) Ligtenberg (AU 2021236529 B2) shows a virtual keyboard system that switches to a virtual gesture mode.
c) MacIsaac (US 2018/0039404 A1) shows a virtual keyboard system that tracks a user’s hand/fingers and confines the input tracking region to a physical keyboard .
d) Adams (US 9104271 B1) shows a virtual keyboard system that tracks hand movement using a tracked glove.
35. Any inquiry concerning this communication or earlier communications from the examiner should be directed to STEVEN PAUL SAX whose telephone number is (571)272-4072. The examiner can normally be reached Monday - Friday, 9:30 - 6:00 Est.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Usmaan Saeed can be reached at 571-272-4046. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/STEVEN P SAX/Primary Examiner, Art Unit 2146
Prosecution Insights