Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 1, 4-6, 8-14, and 16 are amended. Claims 17-20 are newly added. Claims 1-20 are currently under review.
Response to Arguments
Applicant’s arguments with respect to claims 1-20 have been considered but are moot because the new ground of rejection does not rely on the combination of references applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 1, 4-6, 8-9, 12-14, 16-18, and 20 and its dependents (claims 2-3, 7, 10-11, 15, and 19) are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention. The claim limitations indicate “candidate transcriptions” however the specification is completely silent to the same. For the purposes of examination the office will interpret “candidate transcription” simply as “transcription”.
Claim Objections
Claim 20 is objected to because of the following informalities: typographic error. Appropriate correction is required.
Claim 20, line 5: “second word,
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-7 and 9-15 are rejected under 35 U.S.C. 103 as being unpatentable over Thorn (Pub. No.: US 2015/0364140 A1) in view of Marggraff et al. (Pub. No.: US 2017/0123492 A1) hereinafter referred to as Marggraff.
With respect to Claim 1, Thorn teaches a method (fig. 3; ¶80), performed under control of a hardware processor (fig. 1, item 1 comprises item 15: main processor OR fig. 11, item 113 comprises item 115: a processing device; ¶80; ¶123), for interacting with content, comprising: receiving spoken input from a user from a microphone (¶28, “the portable electronic equipment may comprise a microphone”); translating the spoken input into text including a plurality of words (¶123, “Text generated by the speech to text conversion module may be displayed at the wearable device 113”; ¶134); generating a plurality of candidate transcriptions for a portion of the text presented to the user based on a gaze of the user (fig. 4 and fig. 15); causing a wearable display to present the text to the user (figs. 4 and 15; ¶123); based at least on data from a gaze tracking system (fig. 11, item 2: ¶74), receiving a selection of a portion of the text presented to the user based on a gaze of the user (¶96, “a selection of the activation area 41 by eye gaze”; ¶97; ¶124-125; ¶129, “The activation module 128 may activate the text editing function for editing a word or interword space if the heat map data indicates that a dwell time of the user's gaze exceeds a threshold”; ¶135; ¶140-141); presenting the plurality of candidate transcriptions for the selected portion of the text (fig. 4 , item 44; ¶98); receiving a selection of a candidate transcription from the plurality of candidate transcriptions (¶98; ¶136); and replacing the selected portion of the text with the selected candidate transcription (figs. 5 and 16; ¶98; ¶130; ¶137).
Thorn does not mention that the content is virtual content.
Marggraff teaches a method (figs. 22A and 22B; ¶340), performed under control of a hardware processor (¶334), for interacting with virtual content (fig. 27; ¶96; ¶215; ¶220; ¶319, ““Transport” may involve a wide range of displays including virtual reality, augmented reality, tablets, mobile phones, smart devices, projected images, display monitors, signage, and the like”), comprising: receiving spoken input from a user from a microphone (¶44, “a microphone is provided that includes identifying, with the detector, when the user's one or both eyes are directed at a region on the display that contains a plurality of selectable targets … confirming, with the microphone and a data base of sound templates, that the user produces a sound that corresponds to one of the symbols associated with one or more of the selectable targets”); causing a wearable display to present the text to the user (figs. 22A and 22B; ¶325; ¶344, “the acquisition of eye-signals to implement a transport mechanism may be via detectors positioned on or near remote or wearable objects or devices (including head-mounted devices)”); based at least on data from a gaze tracking system (¶341, “Using any combination of eye signals and multi-modal (e.g., mouse, track pad) inputs, a pointer 533a is first moved to the beginning of the sentence where the beginning of the selected text is specified”), receiving a selection of a portion of the text presented to the user based on a gaze of the user (¶341, “By next looking at the region near the end of the sentence 535 and continuing the process of using multi-modal (e.g., mouse, track pad) control, the position of the pointer is “transported” 534 toward the end of the sentence. Movement toward the gaze direction (i.e., end of the sentence 535) may be instantaneous or at a high velocity”; ¶342, “The pointer may (optionally) be shown to slow down as it approaches the gaze location”); and providing the user with an opportunity to edit the portion of the text presented to the user (¶343).
Therefore it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the method of Thorn such that the content is virtual content, as taught by Marggraff, so as to implement the method in a variety of applications.
With respect to Claim 2, claim 1 is incorporated, Thorn teaches wherein receiving the selection of the portion of the text presented to the user comprises determining that the gaze of the user was focused on the portion of the text for at least a predetermined threshold period of time (¶129, “The activation module 128 may activate the text editing function for editing a word or interword space if the heat map data indicates that a dwell time of the user's gaze exceeds a threshold”).
With respect to Claim 3, claim 1 is incorporated, Thorn teaches wherein receiving the selection of the portion of the text presented to the user comprises determining that the gaze of the user is focused on the portion of the text presented to the user (¶76).
Thorn does not teach wherein receiving the selection of the portion of the text presented to the user comprises determining that the gaze of the user is focused on the portion of the text presented to the user while receiving data for an actuation of a user input device.
Marggraff teaches a method (figs. 22A and 22B; ¶340), performed under control of a hardware processor (¶334), for interacting with virtual content (fig. 27; ¶96; ¶215; ¶220; ¶319, ““Transport” may involve a wide range of displays including virtual reality, augmented reality, tablets, mobile phones, smart devices, projected images, display monitors, signage, and the like”), comprising: receiving spoken input from a user from a microphone (¶44, “a microphone is provided that includes identifying, with the detector, when the user's one or both eyes are directed at a region on the display that contains a plurality of selectable targets … confirming, with the microphone and a data base of sound templates, that the user produces a sound that corresponds to one of the symbols associated with one or more of the selectable targets”); causing a wearable display to present the text to the user (figs. 22A and 22B; ¶325; ¶344, “the acquisition of eye-signals to implement a transport mechanism may be via detectors positioned on or near remote or wearable objects or devices (including head-mounted devices)”); based at least on data from a gaze tracking system (¶341, “Using any combination of eye signals and multi-modal (e.g., mouse, track pad) inputs, a pointer 533a is first moved to the beginning of the sentence where the beginning of the selected text is specified”), receiving a selection of a portion of the text presented to the user based on a gaze of the user (¶341, “By next looking at the region near the end of the sentence 535 and continuing the process of using multi-modal (e.g., mouse, track pad) control, the position of the pointer is “transported” 534 toward the end of the sentence. Movement toward the gaze direction (i.e., end of the sentence 535) may be instantaneous or at a high velocity”; ¶342, “The pointer may (optionally) be shown to slow down as it approaches the gaze location”); and providing the user with an opportunity to edit the portion of the text presented to the user (¶343); wherein receiving the selection of the portion of the text presented to the user comprises determining that the gaze of the user is focused on the portion of the text presented to the user while receiving data for an actuation of a user input device (¶341, “Using any combination of eye signals and multi-modal (e.g., mouse, track pad) inputs, a pointer 533a is first moved to the beginning of the sentence where the beginning of the selected text is specified (e.g., by a button press using a mouse)”).
Therefore it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the method of Thorn, wherein receiving the selection of the portion of the text presented to the user comprises determining that the gaze of the user is focused on the portion of the text presented to the user while receiving data for an actuation of a user input device, as taught by Marggraff, so as to precisely specify the end of the sentence (i.e., the end of the cut-and-paste region) with high precision while rapidly traversing large portions of the screen(s), saving considerable time and effort (¶342).
With respect to Claim 4, claim 1 is incorporated, Thorn teaches wherein receiving the selection of the portion of the text presented to the user comprises determining that the gaze of the user is focused on the portion of the text presented to the user (fig. 15, item 151; ¶76; ¶143, “The text editing function may be activated for a region 151 on the display. The activation may be done by eye gaze, as described above, or even by touch”) and substantially while receiving data from a gesture tracking system (eye gaze tracking) indicating that the user made a predetermined command gesture (¶143, eye gaze meets a predefined criterion) requesting presentation of the plurality of candidate transcriptions (fig. 15, item 151; ¶143, “If the dwell time of the user's eye gaze on the region 151 meets a pre-defined criterion, the user may continue to dictate words for insertion at the selected location in the text”).
With respect to Claim 5, claim 1 is incorporated, Thorn teaches wherein receiving the selection of the portion of the text presented to the user comprises determining that the gaze of the user is focused on the portion of the text presented to the user (figs. 15 and 16) and substantially while receiving data from a speech command indicating that the user is requesting presentation of the plurality of candidate transcriptions (¶143, “The text editing function may be activated for a region 151 on the display. The activation may be done by eye gaze, as described above, or even by touch. The user may dictate one word. If the dwell time of the user's eye gaze on the region 151 meets a pre-defined criterion, the user may continue to dictate words for insertion at the selected location in the text”).
With respect to Claim 6, claim 1 is incorporated, Thorn teaches further comprising: based at least one data from the gaze tracking system, receiving a selection of an additional word in the text presented to the user (fig. 4; figs. 15- 16; ¶76, “The text editing function may be allow a user to select from among several candidate words and/or candidate characters which may be selected depending on which word or interword space the user is gazing at”; ¶140); and presenting a plurality of candidate transcriptions for a phrase formed from the portion of the text presented to the user and the additional word (¶98, “various text strings 44 may be displayed by the text editing function from which the user may select one by using his eye gaze. The selected word may replace the word 42”; ¶141-143); receiving a selection of a candidate transcription from the plurality of candidate transcriptions (¶98, “the user may select one by using his eye gaze. The selected word may replace the word 42”); and replacing the phrase with the selected candidate transcription (fig. 4; ¶98, “The selected word may replace the word 42” or fig. 5; ¶102-104).
With respect to Claim 7, claim 1 is incorporated, Thorn teaches wherein at least a portion of the text is emphasized on the wearable display where the portion is associated with a low confidence that a translation from the spoken input to the corresponding portion of the text is correct (fig. 4; ¶20, “The portable electronic equipment may be configured such that the activation area for activating the text editing function for a word covers the pixels on which the word is displayed on the display and an area surrounding the pixels on which the word is displayed on the display. The activation area may be dimensioned in accordance with a resolution of the eye gaze tracking device”; ¶39; ¶86, “the size of an activation area on which the user's eye gaze must be directed for activating the text editing function for a word or interword space may be set depending on a score which quantifies the likelihood of misinterpretation for the particular word and/or a score which quantifies the likelihood that a punctuation mark or other special character is to be inserted at a particular interword space”; ¶96-97; ¶100, “the boundary of the activation area may be displayed”).
With respect to Claim 9, Thorn teaches a system (fig. 1, item 1 OR fig. 11, item 113; ¶80; ¶123) for interacting with content, the system comprising: a display system of a wearable device configured to present the content to a user (¶123, “The wearable device 113 may comprise a display surface at which the text generated by speech to text conversion may be output to the user.”); and a hardware processor (fig. 1, item 1 comprises item 15: main processor OR fig. 11, item 113 comprises item 115: a processing device; ¶80; ¶123) in communication with a sensor (¶28, “the portable electronic equipment may comprise a microphone and/or an Electromyography (EMG) sensor configured to capture speech signals”), a microphone (¶28, “the portable electronic equipment may comprise a microphone”), a gaze tracking system (fig. 11, item 2; ¶74), and the display system (¶123), the hardware processor programmed to: receive spoken input from the user from the microphone (¶28, “the portable electronic equipment may comprise a microphone”); translate the spoken input into text including a plurality of words (¶123, “Text generated by the speech to text conversion module may be displayed at the wearable device 113”); generate a plurality of candidate transcriptions for a portion of the text (fig. 4 and fig. 15); cause the wearable device to present the text to the user (figs. 4 and 15; ¶123); based at least on data from the gaze tracking system (fig. 11, item 2: ¶74), receive a selection of a portion of the text presented to the user based on a gaze of the user (¶96, “a selection of the activation area 41 by eye gaze”; ¶97; ¶124-125; ¶129, “The activation module 128 may activate the text editing function for editing a word or interword space if the heat map data indicates that a dwell time of the user's gaze exceeds a threshold”; ¶135; ¶140-141); present the plurality of candidate transcriptions for the selected portion of the text (fig. 4 , item 44; ¶98); receive a selection of a candidate transcription from the plurality of candidate transcriptions (¶98; ¶136); and replace the selected portion of the text with the selected candidate transcription (figs. 5 and 16; ¶98; ¶130; ¶137).
Thorn does not mention that the content is virtual content.
Marggraff teaches a system (¶344, head-mounted devices) for interacting with virtual content (fig. 27; ¶96; ¶215; ¶220; ¶319, ““Transport” may involve a wide range of displays including virtual reality, augmented reality, tablets, mobile phones, smart devices, projected images, display monitors, signage, and the like”), the system comprising: a display system of a wearable device configured to present the content to a user (¶344); and a hardware processor (¶334) in communication with a microphone (¶44), a gaze tracking system (¶281), and the display system (¶344), the hardware processor programmed to: receive spoken input from a user from a microphone (¶44, “a microphone is provided that includes identifying, with the detector, when the user's one or both eyes are directed at a region on the display that contains a plurality of selectable targets … confirming, with the microphone and a data base of sound templates, that the user produces a sound that corresponds to one of the symbols associated with one or more of the selectable targets”); cause a wearable display to present the text to the user (figs. 22A and 22B; ¶325; ¶344, “the acquisition of eye-signals to implement a transport mechanism may be via detectors positioned on or near remote or wearable objects or devices (including head-mounted devices)”); based at least on data from a gaze tracking system (¶341, “Using any combination of eye signals and multi-modal (e.g., mouse, track pad) inputs, a pointer 533a is first moved to the beginning of the sentence where the beginning of the selected text is specified”), receive a selection of a portion of the text presented to the user based on a gaze of the user (¶341, “By next looking at the region near the end of the sentence 535 and continuing the process of using multi-modal (e.g., mouse, track pad) control, the position of the pointer is “transported” 534 toward the end of the sentence. Movement toward the gaze direction (i.e., end of the sentence 535) may be instantaneous or at a high velocity”; ¶342, “The pointer may (optionally) be shown to slow down as it approaches the gaze location”); and provide the user with an opportunity to edit the portion of the text presented to the user (¶343).
Therefore it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the system of Thorn such that the content is virtual content, as taught by Marggraff, so as to implement the method in a variety of applications.
With respect to Claim 10, claim 9 is incorporated, Thorn teaches wherein the received selection of the portion of the text presented to the user comprises determining that the gaze of the user was focused on the portion of the text for at least a predetermined threshold period of time (¶129, “The activation module 128 may activate the text editing function for editing a word or interword space if the heat map data indicates that a dwell time of the user's gaze exceeds a threshold”).
With respect to Claim 11, claim 9 is incorporated, Thorn teaches wherein the received selection of the portion of the text presented to the user comprises determining that the gaze of the user is focused on the portion of the text presented to the user (¶76).
Thorn does not teach wherein receiving the selection of the portion of the text presented to the user comprises determining that the gaze of the user is focused on the portion of the text presented to the user while receiving data for an actuation of a user input device.
Marggraff teaches a system (¶344, head-mounted devices) for interacting with virtual content (fig. 27; ¶96; ¶215; ¶220; ¶319, ““Transport” may involve a wide range of displays including virtual reality, augmented reality, tablets, mobile phones, smart devices, projected images, display monitors, signage, and the like”), the system comprising: a display system of a wearable device configured to present the content to a user (¶344); and a hardware processor (¶334) in communication with a microphone (¶44), a gaze tracking system (¶281), and the display system (¶344), the hardware processor programmed to: receive spoken input from a user from a microphone (¶44, “a microphone is provided that includes identifying, with the detector, when the user's one or both eyes are directed at a region on the display that contains a plurality of selectable targets … confirming, with the microphone and a data base of sound templates, that the user produces a sound that corresponds to one of the symbols associated with one or more of the selectable targets”); cause a wearable display to present the text to the user (figs. 22A and 22B; ¶325; ¶344, “the acquisition of eye-signals to implement a transport mechanism may be via detectors positioned on or near remote or wearable objects or devices (including head-mounted devices)”); based at least on data from a gaze tracking system (¶341, “Using any combination of eye signals and multi-modal (e.g., mouse, track pad) inputs, a pointer 533a is first moved to the beginning of the sentence where the beginning of the selected text is specified”), receive a selection of a portion of the text presented to the user based on a gaze of the user (¶341, “By next looking at the region near the end of the sentence 535 and continuing the process of using multi-modal (e.g., mouse, track pad) control, the position of the pointer is “transported” 534 toward the end of the sentence. Movement toward the gaze direction (i.e., end of the sentence 535) may be instantaneous or at a high velocity”; ¶342, “The pointer may (optionally) be shown to slow down as it approaches the gaze location”); and provide the user with an opportunity to edit the portion of the text presented to the user (¶343); wherein receiving the selection of the portion of the text presented to the user comprises determining that the gaze of the user is focused on the portion of the text presented to the user while receiving data for an actuation of a user input device (¶341, “Using any combination of eye signals and multi-modal (e.g., mouse, track pad) inputs, a pointer 533a is first moved to the beginning of the sentence where the beginning of the selected text is specified (e.g., by a button press using a mouse)”).
Therefore it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the system of Thorn, wherein receiving the selection of the portion of the text presented to the user comprises determining that the gaze of the user is focused on the portion of the text presented to the user while receiving data for an actuation of a user input device, as taught by Marggraff, so as to precisely specify the end of the sentence (i.e., the end of the cut-and-paste region) with high precision while rapidly traversing large portions of the screen(s), saving considerable time and effort (¶342).
With respect to Claim 12, claim 9 is incorporated, Thorn teaches wherein the received selection of the portion of the text presented to the user comprises determining that the gaze of the user is focused on the portion of the text presented to the user (fig. 15, item 151; ¶76; ¶143, “The text editing function may be activated for a region 151 on the display. The activation may be done by eye gaze, as described above, or even by touch”) and substantially while receiving data from a gesture tracking system (eye gaze tracking) indicating that the user made a predetermined command gesture (¶143, eye gaze meets a predefined criterion) requesting presentation of the plurality of candidate transcriptions (fig. 15, item 151; ¶143, “If the dwell time of the user's eye gaze on the region 151 meets a pre-defined criterion, the user may continue to dictate words for insertion at the selected location in the text”).
With respect to Claim 13, claim 9 is incorporated, Thorn teaches wherein the received selection of the portion of the text presented to the user comprises determining that the gaze of the user is focused on the portion of the text presented to the user (figs. 15 and 16) and substantially while receiving data from a speech command indicating that the user is requesting presentation of the plurality of candidate transcriptions (¶143, “The text editing function may be activated for a region 151 on the display. The activation may be done by eye gaze, as described above, or even by touch. The user may dictate one word. If the dwell time of the user's eye gaze on the region 151 meets a pre-defined criterion, the user may continue to dictate words for insertion at the selected location in the text”).
With respect to Claim 14, claim 9 is incorporated, Thorn teaches wherein the hardware processor is further programmed to: based at least one data from the gaze tracking system, receive a selection of an additional word in the text presented to the user (fig. 4; figs. 15- 16; ¶76, “The text editing function may be allow a user to select from among several candidate words and/or candidate characters which may be selected depending on which word or interword space the user is gazing at”; ¶140); and presenting a plurality of candidate transcriptions for a phrase formed from the portion of the text presented to the user and the additional word (¶98, “various text strings 44 may be displayed by the text editing function from which the user may select one by using his eye gaze. The selected word may replace the word 42”; ¶141-143); receiving a selection of a candidate transcription from the plurality of candidate transcriptions (¶98, “the user may select one by using his eye gaze. The selected word may replace the word 42”); and replacing the phrase with the selected candidate transcription. (fig. 4; ¶98, “to edit the word 42, various text strings 44 may be displayed by the text editing function from which the user may select one by using his eye gaze. The selected word may replace the word 42” or fig. 5; ¶102-104).
With respect to Claim 15, claim 9 is incorporated, Thorn teaches wherein at least a portion of the text is emphasized on the wearable display where the portion is associated with a low confidence that a translation from the spoken input to the corresponding portion of the text is correct (fig. 4; ¶20, “The portable electronic equipment may be configured such that the activation area for activating the text editing function for a word covers the pixels on which the word is displayed on the display and an area surrounding the pixels on which the word is displayed on the display. The activation area may be dimensioned in accordance with a resolution of the eye gaze tracking device”; ¶39; ¶86, “the size of an activation area on which the user's eye gaze must be directed for activating the text editing function for a word or interword space may be set depending on a score which quantifies the likelihood of misinterpretation for the particular word and/or a score which quantifies the likelihood that a punctuation mark or other special character is to be inserted at a particular interword space”; ¶96-97; ¶100, “the boundary of the activation area may be displayed”).
Claims 8 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Thorn and Marggraff as applied to claims 1 and 9 above, and further in view of Chandler et al. (Pub. No.: US 2015/0355805 A1) hereinafter referred to as Chandler.
With respect to Claim 8, claim 1 is incorporated, Thorn teaches wherein the receiving the selection of the portion of the text presented to the user comprises determining that the gaze of the user is focused on the portion of the text presented to the user (fig. 4, item 41; ¶76; ¶97) indicating that the user is requesting presentation of the plurality of candidate transcriptions (fig. 4, item 44; ¶98).
Thorn does not teach wherein the receiving the selection of the portion of the text presented to the user comprises determining that the gaze of the user is focused on the portion of the text presented to the user and substantially while receiving data from a gesture tracking system indicating that the user is requesting presentation of the plurality of candidate transcriptions, and wherein the receiving the data from the gesture tracking system includes determining a type of command gesture from a group of command gestures that includes at least a first command gesture.
Marggraff teaches a method (figs. 22A and 22B; ¶340), performed under control of a hardware processor (¶334), for interacting with virtual content (fig. 27; ¶96; ¶215; ¶220; ¶319, ““Transport” may involve a wide range of displays including virtual reality, augmented reality, tablets, mobile phones, smart devices, projected images, display monitors, signage, and the like”), comprising: receiving spoken input from a user from a microphone (¶44, “a microphone is provided that includes identifying, with the detector, when the user's one or both eyes are directed at a region on the display that contains a plurality of selectable targets … confirming, with the microphone and a data base of sound templates, that the user produces a sound that corresponds to one of the symbols associated with one or more of the selectable targets”); causing a wearable display to present the text to the user (figs. 22A and 22B; ¶325; ¶344, “the acquisition of eye-signals to implement a transport mechanism may be via detectors positioned on or near remote or wearable objects or devices (including head-mounted devices)”); based at least on data from a gaze tracking system (¶341, “Using any combination of eye signals and multi-modal (e.g., mouse, track pad) inputs, a pointer 533a is first moved to the beginning of the sentence where the beginning of the selected text is specified”), receiving a selection of a portion of the text presented to the user based on a gaze of the user (¶341, “By next looking at the region near the end of the sentence 535 and continuing the process of using multi-modal (e.g., mouse, track pad) control, the position of the pointer is “transported” 534 toward the end of the sentence. Movement toward the gaze direction (i.e., end of the sentence 535) may be instantaneous or at a high velocity”; ¶342, “The pointer may (optionally) be shown to slow down as it approaches the gaze location”); and providing the user with an opportunity to edit the portion of the text presented to the user (¶343); wherein the receiving the selection of the portion of the text presented to the user comprises determining that the gaze of the user is focused on the portion of the text presented to the user and substantially while receiving data from a gesture tracking system indicating that the user is requesting to edit (¶330; ¶341, “Using any combination of eye signals and multi-modal (e.g., mouse, track pad) inputs, a pointer 533a is first moved to the beginning of the sentence where the beginning of the selected text is specified (e.g., by a button press using a mouse)”), and wherein the receiving the data from the gesture tracking system includes determining a type of command gesture from a group of command gestures that includes at least a first command gesture (¶330, “the user may “gesture” with the control device that rapid movement toward a gaze location is desired. Gestures may include a rapid movement (i.e., exceeding a threshold velocity), shaking the device (e.g., back-and-forth), one or more patterns of movement (e.g., circular, movement in a fixed direction, etc.)”).
Therefore it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the method of Thorn, such that the user is requesting to edit corresponds to the user is requesting presentation of the plurality of candidate transcriptions resulting in wherein the receiving the selection of the portion of the text presented to the user comprises determining that the gaze of the user is focused on the portion of the text presented to the user and substantially while receiving data from a gesture tracking system indicating that the user is requesting presentation of the plurality of candidate transcriptions, and wherein the receiving the data from the gesture tracking system includes determining a type of command gesture from a group of command gestures that includes at least a first command gesture, as taught by Marggraff, so as to precisely specify the end of the sentence (i.e., the end of the cut-and-paste region) with high precision while rapidly traversing large portions of the screen(s), saving considerable time and effort (¶342).
Thorn and Marggraff combined do not teach wherein the receiving the selection of the portion of the text presented to the user comprises determining that the gaze of the user is focused on the portion of the text presented to the user and substantially while receiving data from a gesture tracking system indicating that the user is requesting an edit, and wherein the receiving the data from the gesture tracking system includes determining a type of command gesture from a group of command gestures that includes at least a first command gesture and a second command gesture, and wherein the first command gesture indicates a first type of replacement option and the second command gesture indicates a second type of replacement option.
Chandler teaches a system and method (¶31; ¶81, “methods and systems for implementing the methods comprising the steps of sensing eye movement via a motion sensor, selecting an object displayed in a display field when the eye pauses at an object for a dwell time sufficient for the motion sensor to detect the pause and simultaneously activating the selected object, repeating the sensing and selecting until the object is either activatable or an attribute capable of direct control”) comprising: receiving input data from two or more user input components comprising an eye gaze tracking device configured to detect eye gaze direction (¶82, “eye tracking using motion for selection/control eye movement is used to select a first word in a sentence of a word document”), and a gesture tracking device configured to acquire data indicating hand gestures of the user (¶82, “Movement of one of the fingers towards the side of the monitor”); determining a selection of a portion of the presented text in the displayed text based at least on the user’s gaze being focused on the portion of the presented text and substantially while receiving data from a gesture tracking system indicating that the user made a predetermined command gesture requesting an edit; and providing the user with an opportunity to edit the portion of the presented text (¶82, “Eye movement is then tracked to the last word in the sentence and another finger (e.g., the left finger) confirms selection. Selected sentence is highlighted due to second motion defining the boundary of selection. The same effect may be had by moving the same finger towards the second eye position (the end of the sentence or word)” and “Movement of one of the fingers towards the side of the monitor (movement is in different direction than the confirmation move) sends a command to delete the sentence. Alternatively, movement of eye to a different location, followed by both fingers moving generally towards that location results in the sentence being copied to the location at which the eyes stopped”); wherein the receiving the selection of the portion of the text presented to the user comprises determining that the gaze of the user is focused on the portion of the text presented to the user and substantially while receiving data from a gesture tracking system indicating that the user is requesting an edit (¶82), and wherein the receiving the data from the gesture tracking system includes determining a type of command gesture from a group of command gestures that includes at least a first command gesture (¶82) and a second command gesture (¶82), and wherein the first command gesture indicates a first type of replacement option and the second command gesture indicates a second type of replacement option (¶82, first type of replacement option: select and highlight, the text is replaced with a highlighted text, second type of replacement option: delete, the text is replaced with a blank space).
Therefore it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the combined method of Thorn and Marggraff, wherein the receiving the selection of the portion of the text presented to the user comprises determining that the gaze of the user is focused on the portion of the text presented to the user and substantially while receiving data from a gesture tracking system indicating that the user is requesting presentation of the plurality of candidate transcriptions, and wherein the receiving the data from the gesture tracking system includes determining a type of command gesture from a group of command gestures that includes at least a first command gesture and a second command gesture, and wherein the first command gesture indicates a first type of replacement option and the second command gesture indicates a second type of replacement option, as taught by Chandler, so as to allow selecting and activating of virtual and/or real objects with ease and accuracy (¶15).
With respect to Claim 16, claim 9 is incorporated, Thorn teaches wherein the received selection of the portion of the text presented to the user comprises determining that the gaze of the user is focused on the portion of the text presented to the user (¶76) indicating that the user is requesting presentation of the plurality of candidate transcriptions (fig. 4, item 44; ¶98).
Thorn does not teach wherein the receiving the selection of the portion of the text presented to the user comprises determining that the gaze of the user is focused on the portion of the text presented to the user and substantially while receiving data from a gesture tracking system indicating that the user is requesting presentation of the plurality of candidate transcriptions, and wherein the receiving the data from the gesture tracking system includes determining a type of command gesture from a group of command gestures that includes at least a first command gesture.
Marggraff teaches a system (¶344, head-mounted devices) for interacting with virtual content (fig. 27; ¶96; ¶215; ¶220; ¶319, ““Transport” may involve a wide range of displays including virtual reality, augmented reality, tablets, mobile phones, smart devices, projected images, display monitors, signage, and the like”), the system comprising: a display system of a wearable device configured to present the content to a user (¶344); and a hardware processor (¶334) in communication with a microphone (¶44), a gaze tracking system (¶281), and the display system (¶344), the hardware processor programmed to: receive spoken input from a user from a microphone (¶44, “a microphone is provided that includes identifying, with the detector, when the user's one or both eyes are directed at a region on the display that contains a plurality of selectable targets … confirming, with the microphone and a data base of sound templates, that the user produces a sound that corresponds to one of the symbols associated with one or more of the selectable targets”); cause a wearable display to present the text to the user (figs. 22A and 22B; ¶325; ¶344, “the acquisition of eye-signals to implement a transport mechanism may be via detectors positioned on or near remote or wearable objects or devices (including head-mounted devices)”); based at least on data from a gaze tracking system (¶341, “Using any combination of eye signals and multi-modal (e.g., mouse, track pad) inputs, a pointer 533a is first moved to the beginning of the sentence where the beginning of the selected text is specified”), receive a selection of a portion of the text presented to the user based on a gaze of the user (¶341, “By next looking at the region near the end of the sentence 535 and continuing the process of using multi-modal (e.g., mouse, track pad) control, the position of the pointer is “transported” 534 toward the end of the sentence. Movement toward the gaze direction (i.e., end of the sentence 535) may be instantaneous or at a high velocity”; ¶342, “The pointer may (optionally) be shown to slow down as it approaches the gaze location”); and provide the user with an opportunity to edit the portion of the text presented to the user (¶343); wherein the received selection of the portion of the text presented to the user comprises determining that the gaze of the user is focused on the portion of the text presented to the user and substantially while receiving data from a gesture tracking system indicating that the user is requesting to edit (¶330; ¶341, “Using any combination of eye signals and multi-modal (e.g., mouse, track pad) inputs, a pointer 533a is first moved to the beginning of the sentence where the beginning of the selected text is specified (e.g., by a button press using a mouse)”), and wherein the received data from the gesture tracking system includes determining a type of command gesture from a group of command gestures that includes at least a first command gesture (¶330, “the user may “gesture” with the control device that rapid movement toward a gaze location is desired. Gestures may include a rapid movement (i.e., exceeding a threshold velocity), shaking the device (e.g., back-and-forth), one or more patterns of movement (e.g., circular, movement in a fixed direction, etc.)”).
Therefore it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the system of Thorn, such that the user is requesting to edit corresponds to the user is requesting presentation of the plurality of candidate transcriptions resulting in wherein the received selection of the portion of the text presented to the user comprises determining that the gaze of the user is focused on the portion of the text presented to the user and substantially while receiving data from a gesture tracking system indicating that the user is requesting presentation of the plurality of candidate transcriptions, and wherein the received data from the gesture tracking system includes determining a type of command gesture from a group of command gestures that includes at least a first command gesture, as taught by Marggraff, so as to precisely specify the end of the sentence (i.e., the end of the cut-and-paste region) with high precision while rapidly traversing large portions of the screen(s), saving considerable time and effort (¶342).
Thorn and Marggraff combined do not teach wherein the receiving the selection of the portion of the text presented to the user comprises determining that the gaze of the user is focused on the portion of the text presented to the user and substantially while receiving data from a gesture tracking system indicating that the user is requesting an edit, and wherein the receiving the data from the gesture tracking system includes determining a type of command gesture from a group of command gestures that includes at least a first command gesture and a second command gesture, and wherein the first command gesture indicates a first type of replacement operation and the second command gesture indicates a second type of replacement operation.
Chandler teaches a system and method (¶31; ¶81, “methods and systems for implementing the methods comprising the steps of sensing eye movement via a motion sensor, selecting an object displayed in a display field when the eye pauses at an object for a dwell time sufficient for the motion sensor to detect the pause and simultaneously activating the selected object, repeating the sensing and selecting until the object is either activatable or an attribute capable of direct control”) comprising: receiving input data from two or more user input components comprising an eye gaze tracking device configured to detect eye gaze direction (¶82, “eye tracking using motion for selection/control eye movement is used to select a first word in a sentence of a word document”), and a gesture tracking device configured to acquire data indicating hand gestures of the user (¶82, “Movement of one of the fingers towards the side of the monitor”); determining a selection of a portion of the presented text in the displayed text based at least on the user’s gaze being focused on the portion of the presented text and substantially while receiving data from a gesture tracking system indicating that the user made a predetermined command gesture requesting an edit; and providing the user with an opportunity to edit the portion of the presented text (¶82, “Eye movement is then tracked to the last word in the sentence and another finger (e.g., the left finger) confirms selection. Selected sentence is highlighted due to second motion defining the boundary of selection. The same effect may be had by moving the same finger towards the second eye position (the end of the sentence or word)” and “Movement of one of the fingers towards the side of the monitor (movement is in different direction than the confirmation move) sends a command to delete the sentence. Alternatively, movement of eye to a different location, followed by both fingers moving generally towards that location results in the sentence being copied to the location at which the eyes stopped”); wherein the received selection of the portion of the text presented to the user comprises determining that the gaze of the user is focused on the portion of the text presented to the user and substantially while receiving data from a gesture tracking system indicating that the user is requesting an edit (¶82), and wherein the receiving the data from the gesture tracking system includes determining a type of command gesture from a group of command gestures that includes at least a first command gesture (¶82) and a second command gesture (¶82), and wherein the first command gesture indicates a first type of replacement operation and the second command gesture indicates a second type of replacement operation (¶82, first type of replacement operation: select and highlight, the text is replaced with a highlighted text, second type of replacement operation: delete, the text is replaced with a blank space).
Therefore it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the combined method of Thorn and Marggraff, wherein the received selection of the portion of the text presented to the user comprises determining that the gaze of the user is focused on the portion of the text presented to the user and substantially while receiving data from a gesture tracking system indicating that the user is requesting presentation of the plurality of candidate transcriptions, and wherein the receiving the data from the gesture tracking system includes determining a type of command gesture from a group of command gestures that includes at least a first command gesture and a second command gesture, and wherein the first command gesture indicates a first type of replacement operation and the second command gesture indicates a second type of replacement operation, as taught by Chandler, so as to allow selecting and activating of virtual and/or real objects with ease and accuracy (¶15).
Claims 17-18 are rejected under 35 U.S.C. 103 as being unpatentable over Thorn and Marggraff as applied to claim 1 above, and further in view of Kushler (Pub. No.: US 2008/0316212 A1).
With respect to Claim 17, claim 1 is incorporated, Thorn and Marggraff combined do not mention wherein the plurality of candidate transcriptions comprises a primary transcription and one or more secondary transcriptions generated for the portion of the text.
Kushler teaches a method (¶39, “The program memory 160 also contains at least one of a text editing system software 164 for recording and restoring the location of a text insertion position according to the method of the present invention”), performed under control of a hardware processor (fig. 1, item 110; ¶39), comprising: receiving spoken input from a user from a microphone (fig. 1, item 141; ¶39, “a microphone 141 is also coupled to the processor so that any spoken input can be received from the user (predominantly for input action recognition systems implementing speech recognition as a method of text input by the user)”); translating the spoken input into text including a plurality of words (fig. 2A; ¶40); generating a plurality of candidate transcriptions for a portion of the text (fig. 2A, item 202; ¶41); receiving a selection of a portion of the text presented to the user (¶41); wherein the plurality of candidate transcriptions comprises a primary transcription (fig. 2A, item 204; ¶41) and one or more secondary transcriptions (fig. 2A, item heart; ¶41) generated for the portion of the text.
Therefore it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the combined method of Thorn and Marggraff, wherein the plurality of candidate transcriptions comprises a primary transcription and one or more secondary transcriptions generated for the portion of the text, as taught by Kushler so as to provide the user with the next-most-closely corresponding alternate textual object for ease of input (¶38).
With respect to Claim 18, claim 17 is incorporated, Thorn and Marggraff combined do not mention wherein the presenting the plurality of candidate transcriptions comprises simultaneously displaying the primary transcription and the one or more secondary transcriptions for user selection.
Kushler teaches a method (¶39, “The program memory 160 also contains at least one of a text editing system software 164 for recording and restoring the location of a text insertion position according to the method of the present invention”), performed under control of a hardware processor (fig. 1, item 110; ¶39), comprising: receiving spoken input from a user from a microphone (fig. 1, item 141; ¶39, “a microphone 141 is also coupled to the processor so that any spoken input can be received from the user (predominantly for input action recognition systems implementing speech recognition as a method of text input by the user)”); translating the spoken input into text including a plurality of words (fig. 2A; ¶40); generating a plurality of candidate transcriptions for a portion of the text (fig. 2A, item 202; ¶41); receiving a selection of a portion of the text presented to the user (¶41); wherein the plurality of candidate transcriptions comprises a primary transcription (fig. 2A, item 204; ¶41) and one or more secondary transcriptions (fig. 2A, item heart; ¶41) generated for the portion of the text; wherein the presenting the plurality of candidate transcriptions comprises simultaneously displaying the primary transcription and the one or more secondary transcriptions for user selection (fig. 2A).
Therefore it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the combined method of Thorn and Marggraff, wherein the presenting the plurality of candidate transcriptions comprises simultaneously displaying the primary transcription and the one or more secondary transcriptions for user selection, as taught by Kushler so as to provide the user with the next-most-closely corresponding alternate textual object for ease of input (¶38).
Claim 19 is rejected under 35 U.S.C. 103 as being unpatentable over Thorn and Marggraff as applied to claim 1 above, and further in view of Caskey et al. (Pub. No.: US 2015/0006170 A1).
With respect to Claim 19, claim 1 is incorporated, Thorn and Marggraff combined do not teach further comprising: determining an automated speech recognition score associated with the portion of the text; and emphasizing the portion of the text based on the automated speech recognition score indicating a reduced likelihood that the portion of the text was correctly transcribed from the spoken input.
Caskey teaches a method (fig. 2; ¶70) comprising: receiving spoken input from a user (fig. 1, item 200; ¶70); real-time recognition (fig. 2, item 210); further comprising: determining an automated speech recognition score associated with the portion of the text (¶37); and emphasizing the portion of the text based on the automated speech recognition score indicating a reduced likelihood that the portion of the text was correctly transcribed from the spoken input (¶31, “it may optionally highlight the questionable text in the user interface and ask the user to correct or confirm automated corrections”; ¶38, “The questionable text can be identified by picking words with a low confidence score”; ¶41, “The present invention can also provide optional, real-time feedback/correction to the user via audio or graphical interface”).
Therefore it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the combined method of Thorn and Marggraff, to further comprise: determining an automated speech recognition score associated with the portion of the text; and emphasizing the portion of the text based on the automated speech recognition score indicating a reduced likelihood that the portion of the text was correctly transcribed from the spoken input, as taught by Caskey so as to a method and structure in which a user's speech is corrected and improved in real-time, in a highly personalized, time-efficient manner (¶12).
Claim 20 is rejected under 35 U.S.C. 103 as being unpatentable over Thorn and Marggraff as applied to claim 1 above, and further in view of Foerster et al. (Pub. No.: US 2016/0364118 A1) hereinafter referred to as Foerster.
With respect to Claim 20, claim 1 is incorporated, Thorn and Marggraff combined do not teach wherein: receiving the selection of the portion of the text comprises receiving a first selection of a first word of the text and a second selection of a second word of the text; and the selected portion of the text comprises a phrase extending from the first word to the second word, wherein the plurality of candidate transcriptions comprises a plurality of speech-recognition hypotheses generated from the spoken input.
Foerster teaches a method (¶140-141 implemented in item 106 of figure 1) comprising: receiving spoken input from a user (¶25); translating the spoken input into text including a plurality of words (¶25); generating a plurality of candidate transcriptions for a portion of the text (¶25-26); display the text to the user (fig. 1; ¶28); receiving a selection of a candidate transcription from the plurality of candidate transcriptions (¶36); wherein: receiving the selection of the portion of the text comprises receiving a first selection of a first word of the text and a second selection of a second word of the text (¶111); and the selected portion of the text comprises a phrase extending from the first word to the second word (¶111), wherein the plurality of candidate transcriptions comprises a plurality of speech-recognition hypotheses generated from the spoken input (¶42-43).
Therefore it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the combined method of Thorn and Marggraff, wherein: receiving the selection of the portion of the text comprises receiving a first selection of a first word of the text and a second selection of a second word of the text; and the selected portion of the text comprises a phrase extending from the first word to the second word, wherein the plurality of candidate transcriptions comprises a plurality of speech-recognition hypotheses generated from the spoken input, as taught by Foerster, so as to allow a user to more easily select hypothesized items that the user may wish to correct (¶3).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Gomez et al. (Patent No.: US 8,223,088 B1) column 3, lines 17-33; column 17, lines 46-65; column 23, lines 10-25.
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DONNA V Bocar whose telephone number is (571)272-0955. The examiner can normally be reached Monday - Friday 8:30am to 5pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amr A Awad can be reached at (571)272-7764. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DONNA V Bocar/ Primary Examiner, Art Unit 2621
