Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Response to Arguments
Applicant's arguments with respect to claims 12-22 have been considered but are moot in view of the new ground(s) of rejection. Applicant’s arguments are directed to the amended subject matter; new prior art is provided below.
Prior art US 20230122824 A1 NARAYANAN; Ajit has been withdrawn in light of the amendments and replaced with new art (Nguyen) to address virtual keyboard on an augmented reality device.
Note: The claims are not directed towards patent ineligible subject matter under 35 U.S.C. 101
Step 1: IS THE CLAIM DIRECTED TO A PROCESS, MACHINE, MANUFACTURE OR COMPOSITION OF MATTER?
Yes
Step 2A.1: IS THE CLAIM DIRECTED TO A LAW OF NATURE, A NATURAL PHENOMENON (PRODUCT OF NATURE) OR AN ABSTRACT IDEA?
No
Step 2A.2: DOES THE CLAIM RECITE ADDITIONAL ELEMENTS THAT INTEGRATE THE JUDICIAL EXCEPTION INTO A PRACTICAL APPLICATION?
Yes, the claims seek to improve eye tracking with multi-modal use of other technology for better accuracy supported by the specification, and reflected by the claims e.g. “[0112] According to a combination of multi-sensing technologies for input data, the accuracy of data input can significantly increase and a time consumed for such data input can be greatly reduced as compared to text input technology capable of using only eye tracking”. In other words, the claims enable the invention to enhance eye tracking for word extraction compared to traditional word extraction techniques or eye tracking alone.
Supported by the following:
In Finjan Inc. v. Blue Coat Systems, Inc., 879 F.3d 1299, 125 USPQ2d 1282 (Fed. Cir. 2018), the claimed invention was a method of virus scanning that scans an application program, generates a security profile identifying any potentially suspicious code in the program, and links the security profile to the application program. 879 F.3d at 1303-04, 125 USPQ2d at 1285-86. The Federal Circuit noted that the recited virus screening was an abstract idea, and that merely performing virus screening on a computer does not render the claim eligible. 879 F.3d at 1304, 125 USPQ2d at 1286. The court then continued with its analysis under part one of the Alice/Mayo test by reviewing the patent' s specification, which described the claimed security profile as identifying both hostile and potentially hostile operations. The court noted that the security profile thus enables the invention to protect the user against both previously unknown viruses and “obfuscated code,” as compared to traditional virus scanning, which only recognized the presence of previously-identified viruses. The security profile also enables more flexible virus filtering and greater user customization. 879 F.3d at 1304, 125 USPQ2d at 1286. The court identified these benefits as improving computer functionality, and verified that the claims recite additional elements (e.g., specific steps of using the security profile in a particular way) that reflect this improvement. Accordingly, the court held the claims eligible as not being directed to the recited abstract idea. 879 F.3d at 1304-05, 125 USPQ2d at 1286-87. This analysis is equivalent to the Office' s analysis of determining that the additional elements integrate the judicial exception into a practical application at Step 2A Prong Two, and thus that the claims were not directed to the judicial exception (Step 2A: NO).
Examples of claims that improve technology and are not directed to a judicial exception include: Enfish, LLC v. Microsoft Corp., 822 F.3d 1327, 1339, 118 USPQ2d 1684, 1691-92 (Fed. Cir. 2016) (claims to a self-referential table for a computer database were directed to an improvement in computer capabilities and not directed to an abstract idea); McRO, Inc. v. Bandai Namco Games Am. Inc., 837 F.3d 1299, 1315, 120 USPQ2d 1091, 1102-03 (Fed. Cir. 2016) (claims to automatic lip synchronization and facial expression animation were directed to an improvement in computer-related technology and not directed to an abstract idea); Visual Memory LLC v. NVIDIA Corp., 867 F.3d 1253,1259-60, 123 USPQ2d 1712, 1717 (Fed. Cir. 2017) (claims to an enhanced computer memory system were directed to an improvement in computer capabilities and not an abstract idea); Finjan Inc. v. Blue Coat Systems, Inc., 879 F.3d 1299, 125 USPQ2d 1282 (Fed. Cir. 2018) (claims to virus scanning were found to be an improvement in computer technology and not directed to an abstract idea); SRI Int' l, Inc. v. Cisco Systems, Inc., 930 F.3d 1295, 1303 (Fed. Cir. 2019) (claims to detecting suspicious activity by using network monitors and analyzing network packets were found to be an improvement in computer network technology and not directed to an abstract idea). Additional examples are provided in MPEP § 2106.05(a).
Regarding the December 5th 2025 Memo in light of September 26, 2025 Appeals Review Panel Decision in Ex parte Desjardins, Appeal 2024-000567 for Application 16/319,040, in deciding if a recited abstract idea does or does not direct the entire claim to an abstract idea, when a claim is considered as a whole:
Paragraph 21 of the Specification, which the Appellant cites, identifies improvements in training the machine learning model itself. Of course, such an assertion in the Specification alone is insufficient to support a patent eligibility determination, absent a subsequent determination that the claim itself reflects the disclosed improvement. See MPEP § 2106.05(a) (citing Intellectual Ventures I LLC v. Symantec Corp., 838 F.3d 1307, 1316 (Fed. Cir. 2016)). Here, however, we are persuaded that the claims reflect such an improvement. For example, one improvement identified in the 8 Appeal2024-000567 Application 16/319,040 Specification is to "effectively learn new tasks in succession whilst protecting knowledge about previous tasks." Spec. ,r 21. The Specification also recites that the claimed improvement allows artificial intelligence (AI) systems to "us[e] less of their storage capacity" and enables "reduced system complexity." Id. When evaluating the claim as a whole, we discern at least the following limitation of independent claim 1 that reflects the improvement: "adjust the first values of the plurality of parameters to optimize performance of the machine learning model on the second machine learning task while protecting performance of the machine learning model on the first machine learning task." We are persuaded that constitutes an improvement to how the machine learning model itself operates, and not, for example, the identified mathematical calculation. Under a charitable view, the overbroad reasoning of the original panel below is perhaps understandable given the confusing nature of existing § 101 jurisprudence, but troubling, because this case highlights what is at stake. Categorically excluding AI innovations from patent protection in the United States jeopardizes America's leadership in this critical emerging technology. Yet, under the panel's reasoning, many AI innovations are potentially unpatentable-even if they are adequately described and nonobvious-because the panel essentially equated any machine learning with an unpatentable "algorithm" and the remaining additional elements as "generic computer components," without adequate explanation. Dec. 24. Examiners and panels should not evaluate claims at such a high level of generality.
Specifically, Ex Parte Desjardins explained the following:
Enfish ranks among the Federal Circuit's leading cases on the eligibility of technological improvements. In particular, Enfish recognized that “[m]uch of the advancement made in computer technology consists of improvements to software that, by their very nature, may not be defined by particular physical features but rather by logical structures and processes.” 822 F.3d at 1339. Moreover, because “[s]oftware can make non-abstract improvements to computer technology, just as hardware improvements can,” the Federal Circuit held that the eligibility determinations should turn on whether “the claims are directed to an improvement to computer functionality versus being directed to an abstract idea.” Id. at 1336. (Desjardins, page 8).
Further in Ex Parte Desjardins, Appeal No. 2024-000567 (PTAB September 26, 2025, Appeals Review Panel Decision) (precedential), the claimed invention was a method of training a machine learning model on a series of tasks. The Appeals Review Panel (ARP) overall credited benefits including reduced storage, reduced system complexity and streamlining, and preservation of performance attributes associated with earlier tasks during subsequent computational tasks as technological improvements that were disclosed in the patent application specification. Specifically, the ARP upheld the Step 2A Prong One finding that the claims recited an abstract idea (i.e., mathematical concept). In Step 2A Prong Two, the ARP then determined that the specification identified improvements as to how the machine learning model itself operates, including training a machine learning model to learn new tasks while protecting knowledge about previous tasks to overcome the problem of “catastrophic forgetting” encountered in continual learning systems. Importantly, the ARP evaluated the claims as a whole in discerning at least the limitation “adjust the first values of the plurality of parameters to optimize performance of the machine learning model on the second machine learning task while protecting performance of the machine learning model on the first machine learning task” reflected the improvement disclosed in the specification. Accordingly, the claims as a whole integrated what would otherwise be a judicial exception instead into a practical application at Step 2A Prong Two, and therefore the claims were
The claim itself does not need to explicitly recite the improvement described in the specification (e.g., “thereby increasing the bandwidth of the channel”). See, e.g., Ex Parte Desjardins, Appeal No. 2024-000567 (PTAB September 26, 2025, Appeals Review Panel Decision) (precedential), in which the specification identified the improvement to machine learning technology by explaining how the machine learning model is trained to learn new tasks while protecting knowledge about previous tasks to overcome the problem of “catastrophic forgetting,” and that the claims reflected the improvement identified in the specification. Indeed, enumerated improvements identified in the Desjardins specification included disclosures of the effective learning of new tasks in succession in connection with specifically protecting knowledge concerning previously accomplished tasks; allowing the system to reduce use of storage capacity; and the enablement of reduced complexity in the system. Such improvements were tantamount to how the machine learning model itself would function in operation and therefore not subsumed in the identified mathematical calculation.
Step 2B: DOES THE CLAIM RECITE ADDITIONAL ELEMENTS THAT AMOUNT TO SIGNIFICANTLY MORE THAN THE JUDICIAL EXCEPTION?
If Yes at step 2A.1 and step 2A.2 fails, the interpretation in the context of 35 USC 101 amounts to a physical hardware that is not generically cited as a whole, such as eye tracking hardware requiring a camera in conjunction with lip tracking, in which a human in general may not be able to interpret letters being spoken or substitute a computer for eye tracking outside of general observation. In other words, a hardware driven modification of conventional tracking produces significantly more i.e. multimodal dual use tracking of letters and words thereof. A human would not seek claim limitations as a general or routine step, wherein such steps improve the functioning of a tracking and word extraction system by enhancing eye tracking for word extraction compared to traditional word extraction techniques or eye tracking alone or similarly such a technology and its functionality thereof with the specific limitations recited. Such claim language provides significance beyond mere data manipulation or mental decision making and is also analogous to interim amendment examples. Additionally, if one or ordinary skill in the art fails to properly consider any specified hardware or arranged components in the claims, considers such claim limitations non-generic, or simply disregards Enfish, the claims still demonstrate that there exists improvements to the functioning of a computer (or technology), e.g., a modification of conventional Internet hyperlink protocol to dynamically produce a dual-source hybrid webpage, as discussed in DDR Holdings, LLC v. Hotels.com, L.P., 773 F.3d 1245, 1258-59, 113 USPQ2d 1097, 1106-07 (Fed. Cir. 2014) (see MPEP § 2106.05(a));. Therefore, in light of the above as a whole, it is not warranted to give a rejection under 35 USC 101.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
NOTE: The claims are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The claim limitations, specifically claims 12-15 and 19-21 in question recite:
“voice pickup sensor”
“eye tracker”
“lip shape tracker”
In the scope of software-hardware, such elements preceding “tracker” or “sensor” provide structure that aligns with the courts and is analogous to BRI examples such as a “digital detector” in the scope of computing per se, and a “knife blade” unit for cutting, in the scope of non-computing technology. Specifically considering the following:
E. “Detector”
1. Personalized Media Commc’ns, LLC, v. ITC, 161 F.3d 696 (Fed. Cir. 1998)
United States Patent 5,335,277 (“the ’277 patent”)
The claimed subject matter relates to an integrated system for communicating programming, e.g. electronically transmitted entertain, instruct or inform, including television, radio, broadcast print, etc. The relevant claim language of independent claim 44 of the ’277 patent is as follows:
“. . .a digital detector operatively connected to a mass medium receiver for detecting digital information in a mass medium transmission and transferring some of said detected information to a processor; . . .”
Issue: Does the claim limitation “digital detector” invoke 35 U.S.C. § 112, sixth paragraph?
Analysis: The claim limitation does not use “means for” language to invoke 35 U.S.C. § 112, sixth paragraph. To one of ordinary skill in the relevant art, the term “detector” connotes or describes in general a structure. The claim term “digital detector” is subsequently modified by the functional language “for detecting.” The fact that the term “detector” does not suggest a precise physical structure would not result in 35 U.S.C. § 112, sixth paragraph being invoked. The claim term “detector” is understood in the relevant prior art and defined in dictionaries as having a well known meaning in the electrical arts connotative of structure. Further, just because the claim term “detector” is a name for structure drawn from the function it performs, should not result in treatment under 35 U.S.C. 112, sixth paragraph. Therefore the term “detector” is structural and not a nonce word or a verbal construct that is not recognized as the name of structure and simply a substitute for the term “means for.” Accordingly, the presence of a structural term combined with the absence of any “means for” language indicates that 35 U.S.C. § 112, sixth paragraph is not invoked.
Conclusion: 35 U.S.C. § 112, sixth paragraph is not invoked.
The above analysis is in accordance with MPEP 2181 I 8th Ed. Rev. 6., Sept 2007 Pages 2100-236, and further in view of the analytical framework of the 2011 Supplementary Guidelines to determine whether a limitation invokes 35 U.S.C. § 112, sixth paragraph.
And additionally in view of the courts regarding “means” per se, the example of an Ink jet means for ink delivery modified by “ink jet” which is sufficient structure for achieving specified functions. If such elements e.g. unit, module, sensor, were recited on its own in the current claims, such interpretation would not be applicable, and instead a generic placeholder would be present, such as the sole mention of “device” or “apparatus” on its own would in fact invoke 112(f). In the case above, this is not reasonable for the field of software/hardware.
Such claims are believed to not exhibit:
1) a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function, and
2) “means” or “step”, and 2) usage of the word “means” or “step”.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 12-22 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 20180336191 A1 Rao; Ashwin P. (hereinafter Rao) in view of US 20210399911 A1 Jorasch; James et al. (hereinafter Jorasch) and further in view of US 20200218488 A1 Nguyen; Quynh et al. (hereinafter Nguyen).
Re claim 12, Rao teaches
a voice pickup sensor configured to confirm an input of at least one letter; (voice sensor to detect a word or letters e.g. ASR 0033 with fig. 2, with fig. 8 and 0029 where a user enters letters while simultaneously speaking a word where a letter is part of a word or alternatively a user utters a symbol which is a single character)
an eye tracker comprising at least one camera, wherein the eye tracker is configured to detect eye movement of a user through the at least one camera; (eye tracking which requires the use of a camera or web-cam 0035 with fig. 2)
a lip shape tracker configured to infer the at least one letter; and (lip tracking in time, to detect a word or letters 0027 with 0039 & fig. 2)
a processor configured to complete a word based on the inferred at least one letter. (AI driven display of the word/letter based on the best group of fused multi-modal inputs on a time scale 0056 with fig. 1 and fig. 2… using at least voice sensor to detect a word or letters e.g. ASR 0033, eye tracking which requires the use of a camera or web-cam 0035, and lip tracking in time, to detect a word or letters 0027 with 0039)
However, while Rao teaches multi-modal fusion to detect a word or letter with a use in virtual reality, it fails to teach:
12. (New) An augmented reality (AR) device comprising: (Jorasch augmented reality 0219)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Rao to incorporate the above claim limitations as taught by Jorasch to allow for a simple substitution of one known element of recognition of word/letter inputs in a virtual reality context for another of Jorasch’s specific use of augmented reality to obtain predictable results where a user can benefit from the ability to not be isolated in virtual reality such as in a work environment where a user can have augmented reality to have supplemental information e.g. multi-modal fusion of inputs to verify an intent to reduce error thus enhancing both the virtual and real world.
However, while the combination teaches a multi-modal fusion and eye gaze where touch inputs of letters are synched with a word comprising letters from a user such as the start letter of a word, it is silent with respect to a virtual keyboard as precisely claimed:
wherein the display is configured to output a letter input device, the letter input device including a virtual keyboard, and a pointer on the letter input device based on the detected eye movement of the user. (Nguyen using a virtual keyboard in augmented reality on a windshield for instance 0048, eye-gaze 0052 and character selection on a virtual keyboard 0055 in conjunction with ASR 0090)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Rao in view of Jorasch to incorporate the above claim limitations as taught by Nguyen to allow for combining prior art elements according to known methods to yield predictable results such as, where the existing augmented reality of Jorasch and the multi-fusion of Rao is analogous to that of Nguyen, with the addition of a virtual keyboard per se in place of or in addition to a touch screen or on screen keyboard, such that a user can have hands-free and screen free per see while viewing past an object e.g. through glass or on a HUD.
Re claim 13, Rao teaches
13. (New) The AR device according to claim 12, wherein: the voice pickup sensor is further configured to confirm the input of the at least one letter… (AI driven display of the word/letter based on the best group of fused multi-modal inputs on a time scale 0056 with fig. 1 and fig. 2… using at least voice sensor to detect a word or letters e.g. ASR 0033, eye tracking which requires the use of a camera or web-cam 0035, and lip tracking in time, to detect a word or letters 0027 with 0039)
However, while Rao teaches multi-modal fusion to detect a word or letter, it fails to teach:
…based on bone conduction caused by movement of a skull-jaw joint of the user. (Jorasch bone conduction to help hearing impaired uses such as a device which would rest behind the ear above the jaw but on the skull per se 02152-2153)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Rao to incorporate the above claim limitations as taught by Jorasch to allow for combining prior art elements according to known methods to yield predictable results, such as using another mode of listening as a non-invasive option for impaired users providing improved recognition of audio, where such technology is highly effective for speech recognition in noisy environments because bone-conducted microphones (BCMs) are less susceptible to background noise than traditional air-conduction microphones, such that the fusion in Rao can make up for the low quality of the BCM and still provide an otherwise impaired user a reliable option of input.
Re claim 14, Rao teaches multi-model fusion with eye tracking requiring a camera per se, but fails to teach:
14. (New) The AR device according to claim 13, wherein:
the lip shape tracker comprises an infrared (IR) camera and an IR illuminator; and (Jorasch IR + camera with illuminator 0239 with 2537 and 2702)
the lip shape tracker is further configured to infer the at least one letter through the IR camera and the IR illuminator. (Jorasch 0903 and 2041 lip movement, directions thereof, and utilizing IR + camera with illuminator 0239 with 2537 and 2702)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Rao to incorporate the above claim limitations as taught by Jorasch to allow for combining prior art elements according to known methods to yield predictable results, such as utilizing IR and an accompanying illuminator to handle night or low light circumstances when a user would be communicating, thereby maintaining eye tracking as well as lip tracking including silent speech at night to improve fusion for any light condition.
Re claim 15, Rao teaches
15. (New) The AR device according to claim 14, wherein: the lip shape tracker is further configured to infer the at least one letter based on a time taken for the eye tracker to sense the eye movement of the user. (as in fig. 2 the eye tracking is time based utilizing the accumulation of eye movement in a given time to account for a word comprising letters, and further using AI driven fusion of the word/letter based on the best group of fused multi-modal inputs on a time scale 0056 with fig. 1 and fig. 2… using at least voice sensor to detect a word or letters e.g. ASR 0033, eye tracking which requires the use of a camera or web-cam 0035, and lip tracking in time, to detect a word or letters 0027 with 0039)
Re claim 16, Rao teaches multi-model fusion with eye tracking requiring a camera per se, but fails to teach:
16. (New) The AR device according to claim 15, wherein: the IR camera and the IR illuminator are positioned to photograph lips of the user at a preset angle. (Jorasch 0903 and 2041 lip movement, directions thereof, and utilizing IR + camera with illuminator 0239 with 2537 and 2702)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Rao to incorporate the above claim limitations as taught by Jorasch to allow for combining prior art elements according to known methods to yield predictable results, such as utilizing IR and an accompanying illuminator to handle night or low light circumstances when a user would be communicating, thereby maintaining eye tracking as well as lip tracking including silent speech at night to improve fusion for any light condition.
Re claim 17, Rao teaches a voice sensor to detect a word or letters e.g. ASR 0033 with fig. 2, with fig. 8 and 0029 where a user enters letters while simultaneously speaking a word where a letter is part of a word or alternatively a user utters a symbol which is a single character
However, while the combination teaches a multi-modal fusion and eye gaze where touch inputs of letters are synched with a word comprising letters from a user such as the start letter of a word, it is silent with respect to a virtual keyboard in conjunction with spoken input, as precisely claimed:
17. (Currently Amended) The AR device according to claim 16, further configured to output a specific letter of the virtual keyboard in response to detecting both an utterance of the user of the specific letter and a gaze of the user directed to the specific letter on the virtual keyboard. (Nguyen user is speaking an utterance which contains letters of a word per se, intent of characters input using a virtual keyboard in augmented reality on a windshield for instance 0048, eye-gaze 0052 and character selection on a virtual keyboard 0055 in conjunction with ASR 0090)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Rao in view of Jorasch to incorporate the above claim limitations as taught by Nguyen to allow for combining prior art elements according to known methods to yield predictable results such as, where the existing augmented reality of Jorasch and the multi-fusion of Rao is analogous to that of Nguyen, with the addition of a virtual keyboard per se in place of or in addition to a touch screen or on screen keyboard, such that a user can have hands-free and screen free per see while viewing past an object e.g. through glass or on a HUD.
Re claim 18, Rao teaches
18. (New) The AR device according to claim 17, wherein: the display is further configured to output the completed word obtained through the processor. (AI driven fusion and display of the word/letter based on the best group of fused multi-modal inputs on a time scale 0056 with fig. 1 and fig. 2… using at least voice sensor to detect a word or letters e.g. ASR 0033, eye tracking which requires the use of a camera or web-cam 0035, and lip tracking in time, to detect a word or letters 0027 with 0039)
Re claim 19, Rao teaches
19. (New) The AR device according to claim 12, further comprising: an input device, wherein the voice pickup sensor is further configured to start confirmation of the input of the at least one letter based on a control signal received through the input device. (when a user begins speaking for instance or a letter or word is detected on multiple modes, using AI driven fusion and display of the word/letter based on the best group of fused multi-modal inputs on a time scale 0056 with fig. 1 and fig. 2… using at least voice sensor to detect a word or letters e.g. ASR 0033, eye tracking which requires the use of a camera or web-cam 0035, and lip tracking in time, to detect a word or letters 0027 with 0039)
Re claim 20, Rao teaches
20. (New) The AR device according to claim 12, further comprising: a memory device, wherein the lip shape tracker is further configured to infer the at least one letter based on a database stored in the memory device. (AI and model-based e.g. 0027, for AI driven fusion and display of the word/letter based on the best group of fused multi-modal inputs on a time scale 0056 with fig. 1 and fig. 2… using at least voice sensor to detect a word or letters e.g. ASR 0033, eye tracking which requires the use of a camera or web-cam 0035, and lip tracking in time, to detect a word or letters 0027 with 0039)
Re claim 21, Rao teaches
21. (New) The AR device according to claim 12, wherein: the lip shape tracker is further configured to infer the at least one letter using artificial intelligence (AI). (AI driven fusion and display of the word/letter based on the best group of fused multi-modal inputs on a time scale 0056 with fig. 1 and fig. 2… using at least voice sensor to detect a word or letters e.g. ASR 0033, eye tracking which requires the use of a camera or web-cam 0035, and lip tracking in time, to detect a word or letters 0027 with 0039)
Re claim 22, Rao teaches
detecting, via a camera, eye movement of the user; (eye tracking which requires the use of a camera or web-cam 0035 with fig. 2)
completing a word based on the inferred at least one letter. (AI driven fusion and display of the word/letter based on the best group of fused multi-modal inputs on a time scale 0056 with fig. 1 and fig. 2… using at least voice sensor to detect a word or letters e.g. ASR 0033, eye tracking which requires the use of a camera or web-cam 0035, and lip tracking in time, to detect a word or letters 0027 with 0039… and voice sensor to detect a word or letters e.g. ASR 0033 with fig. 2, with fig. 8 and 0029 where a user enters letters while simultaneously speaking a word where a letter is part of a word or alternatively a user utters a symbol which is a single character)
However, while Rao teaches multi-modal fusion to detect a word or letter, it fails to teach:
22. (New) A method for controlling an augmented reality (AR) device, the method comprising: (Jorasch augmented reality 0219)
confirming an input of at least one letter based on bone conduction caused by movement of a skull-jawjoint of a user; (Jorasch bone conduction to help hearing impaired uses such as a device which would rest behind the ear above the jaw but on the skull per se 02152-2153)
inferring, via an infrared (IR) camera and an IR illuminator, the at least one letter; and (Jorasch 0903 and 2041 lip movement, directions thereof, and utilizing IR + camera with illuminator 0239 with 2537 and 2702)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Rao to incorporate the above claim limitations as taught by Jorasch to allow for a simple substitution of one known element of recognition of word/letter inputs in a virtual reality context for another of Jorasch’s specific use of augmented reality to obtain predictable results where a user can benefit from the ability to not be isolated in virtual reality such as in a work environment where a user can have augmented reality to have supplemental information e.g. multi-modal fusion of inputs to verify an intent to reduce error thus enhancing both the virtual and real world, and regarding bon conduction: to allow for combining prior art elements according to known methods to yield predictable results, such as using another mode of listening as a non-invasive option for impaired users providing improved recognition of audio, where such technology is highly effective for speech recognition in noisy environments because bone-conducted microphones (BCMs) are less susceptible to background noise than traditional air-conduction microphones, such that the fusion in Rao can make up for the low quality of the BCM and still provide an otherwise impaired user a reliable option of input, and further, regarding IR concepts: to allow for combining prior art elements according to known methods to yield predictable results, such as utilizing IR and an accompanying illuminator to handle night or low light circumstances when a user would be communicating, thereby maintaining eye tracking as well as lip tracking including silent speech at night to improve fusion for any light condition.
However, while the combination teaches a multi-modal fusion and eye gaze where touch inputs of letters are synched with a word comprising letters from a user such as the start letter of a word, it is silent with respect to a virtual keyboard as precisely claimed:
And outputting a letter input device, the letter input device including a virtual keyboard, and a pointer on the letter input device based on the detected eye movement of the user. (Nguyen using a virtual keyboard in augmented reality on a windshield for instance 0048, eye-gaze 0052 and character selection on a virtual keyboard 0055 in conjunction with ASR 0090)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Rao in view of Jorasch to incorporate the above claim limitations as taught by Nguyen to allow for combining prior art elements according to known methods to yield predictable results such as, where the existing augmented reality of Jorasch and the multi-fusion of Rao is analogous to that of Nguyen, with the addition of a virtual keyboard per se in place of or in addition to a touch screen or on screen keyboard, such that a user can have hands-free and screen free per see while viewing past an object e.g. through glass or on a HUD.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
US 20020135618 A1 Maes, Stephane Herman et al.
Multi-modal inputs
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MICHAEL COLUCCI whose telephone number is (571)270-1847. The examiner can normally be reached on M-F 9 AM - 7 PM.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Flanders can be reached at (571)272-7516. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHAEL COLUCCI/Primary Examiner, Art Unit 2655 (571)-270-1847
Examiner FAX: (571)-270-2847
Michael.Colucci@uspto.gov
Prosecution Insights