Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over CHEN et al (Air-writing recognition—Part I: Modeling and recognition of characters, words, and connecting motions) in view of GAN et al (In-air handwritten Chinese text recognition with temporal convolutional recurrent network).
As per claim 1, Chen teaches the claimed “system for performing three-dimensional optical character recognition, wherein the system comprises: a display, memory, and input and output devices, each communicatively coupled to a processor” (Chen, VI. USABILITY STUDY, A. Methods - The user sits in a living-room like environment with a 65-in full HD display and a handheld tracking device modified from the Wii Remote. The remote control functions as a mouse, with translation in the vertical plane mapped to the cursor movement on the display with a scale of one meter to 2000 pixels… For the virtual keyboard, we use the built-in on-screen keyboard of Windows 7, which is resized to the lower half of the screen. The window of our logger is on the upper half of the screen, and the status box now shows the input letters), wherein the processor is configured to: “determine a plane in an extended reality environment” (Chen, V. MOTION CHARACTER AND MOTION WORD RECOGNITION EVALUATION - A motion gesture can be defined in a 3-D space, but hand writing is actually defined on a 2-D surface regardless of the true writing motions… We can consider P^ and V^ as representing the writing motion projected on a vertical plane; VI. USABILITY STUDY, A. Methods - The remote control functions as a mouse, with translation in the vertical plane mapped to the cursor movement on the display with a scale of one meter to 2000 pixels); “upon determining the plane, modify a location of one or more pixels representing text such that the one or more pixels are located on the plane in the extended reality environment” (Chen, V. MOTION CHARACTER AND MOTION WORD RECOGNITION EVALUATION - A motion gesture can be defined in a 3-D space, but hand writing is actually defined on a 2-D surface regardless of the true writing motions… We can consider P^ and V^ as representing the writing motion projected on a vertical plane); “identify, via a three-dimensional optical character recognition algorithm, one or more characters of the text from the one or more pixels in the extended reality environment” (Chen, C. Word-Based Motion Word Recognition - In the word-based decoding network, each path is a word model synthesized from corresponding character and ligature HMMs, and the letter sequences are tightly restricted to the vocabulary; D. Letter-Based Motion Word Recognition - We choose the refined HMMs of character and decision-tree clustered ligatures to build the letter-based decoding word network). Although Chen does not explicitly teach, but Gan teaches “generate interactable text based on the identified one or more characters from the text; and align the interactable text on the plane in the extended reality environment such that the interactable text is a modifiable representation of the text as presented by the one or more pixels” (Gan, 3.2.1. Pre-processing of in-air handwriting trajectories - Considering the fact that characters are defined as the 2- dimensional symbols, the depth information of handwriting trajectories actually is meaningless and useless for representing characters; instead, the x -, y -coordinates should be the most discriminative information for the final recognition. Moreover, even in the 3D space, people always attempt to write texts in an imaginary 2D surface. Therefore, we project each 3D in-air hand- written text into an optimal 2D plane with the minimum mean square error criterion (as shown in Fig. 7) while discarding the depth information. After that, the projected 2D trajectory is further normalized with the slope correction, size normalization, and re-sampling (as shown in Fig. 8). Those normalization steps are important for recognition since handwriting samples suffer great variations in sizes and orientations). Thus, it would have been obvious, in view of Gan to configure Chen’s method as claimed by modifying the projected/recognized text into a “standard” form for the purpose to representing the text in an enhanced visualization.
Claim 2 adds into claim 1 “determine if the one or more pixels representing the text when the text is already located on or next to a vertical plane of the determined plane in the extended reality environment, wherein the text is representative of one or more words, characters, numbers, symbols, or any combination thereof; and refrain from adjusting the one or more pixels representing the text when the text is already located on or next to a vertical plane of the determined plane in the extended reality environment” which is obvious (i.e., no projection is needed when the text is already located on the determined plane) as in Chen’s case in which the selected plane for projecting the 3D character path is the vertical plane (Chen, V. MOTION CHARACTER AND MOTION WORD RECOGNITION EVALUATION - A motion gesture can be defined in a 3-D space, but hand writing is actually defined on a 2-D surface regardless of the true writing motions… We can consider P^ and V^ as representing the writing motion projected on a vertical plane; VI. USABILITY STUDY, A. Methods - The remote control functions as a mouse, with translation in the vertical plane mapped to the cursor movement on the display with a scale of one meter to 2000 pixels).
Claim 3 adds into claim 1 “receive one or more commands via an input device, wherein the one or more commands are representative of one or more modifications to one or more characteristics of the interactable text; identify the one or more modifications from the one or more commands; and modify, based on the one or more modifications, one or more characteristics of the interactable text” (Chen, VI. Usability Study, A. Methods - The time and traverse distance while holding Button B (writing) is recorded separately. The status box displays the recognized word or input status, e.g., writing or recognizing. If an error occurs, the subject re-writes until the word is correctly recognized, and we accumulate the writing time of each trial; III. AIR-WRITING WITH SIX-DEGREE-OF-FREEDOM MOTION TRACKING - Similar to motion gestures, air-writing is tracked with a continuous stream of sensor data, and the writing is intuitively rendered in the air in unistroke without any pen-up and pen-down information… The system tracks a specially designed handheld device and provides both explicit (position and orientation) and implicit (acceleration and angular speed) 6-DOF data sampled at 60 Hz).
Claim 4 adds into claim 3 “wherein the one or more characteristics comprise size, pitch, yawl, roll, a relative position in the extended virtual reality environment based on the set of coordinates, or any combination thereof” (Chen, Abstract – recognition of characters or words is accomplished based on six-degree-of freedom hand motion data); and “the one or more commands include at least one of a hand movement, analog button input, voice command, and virtual selection via an interactable button in the extended reality environment” (Chen, VI. Usability Study, A. Methods - The time and traverse distance while holding Button B (writing) is recorded separately. The status box displays the recognized word or input status, e.g., writing or recognizing. If an error occurs, the subject re-writes until the word is correctly recognized, and we accumulate the writing time of each trial).
Claim 5 adds into claim 1 “wherein configuration of the processor to determine the plane is based at least upon a minimized standard deviation” (Gan, 3.2.1. Pre-processing of in-air handwriting trajectories - Moreover, even in the 3D space, people always attempt to write texts in an imaginary 2D surface. Therefore, we project each 3D in-air hand- written text into an optimal 2D plane with the minimum mean square error criterion (as shown in Fig. 7) while discarding the depth information). Thus, it would have been obvious, in view of Gan, to configure Chen’s method as claimed by using a statistical model (i.e., minimization of the standard deviation) for defining a 2D plane based on the samples collected on the 3D space. The motivation is to define a written plane of the characters based on the samples of a 3D path of characters.
Claim 6 adds into claim 1 “wherein the processor is configured to generate an accuracy rating based on a comparison of the text and the interactable text” (Chen, VI. USABILITY STUDY, B. Results - Because air writing is recognized on a word basis, we report the average number of attempts to correctly input a word. Longer words tend to have higher recognition accuracy and hence need fewer attempts… Based on our study, air-writing may not be fast enough for general-purpose text input, but it is suitable for infrequent and short text input on a motion-based user interface, where conventional writing or typing is not available).
As per claim 7, Chen teaches the claimed “method,” comprising: “identifying a set of text from three-dimensional writings in an extended reality environment” (Chen, IV. AIR-WRITING PROCESSING AND MODELING, B. Air-Writing Modeling - HMMs of motion characters are trained directly from the isolated A-to-Z recording, and we create one model for each character… For example, the most complicated letter E has 18 states, and short letters I and J have only eight states. These motion character models are the building blocks for motion words); “determining a plane in the extended reality environment based on a set of locations of each text of the set of text, wherein the set of locations comprises a three-dimensional position of each text of the set of text in the extended reality environment” (Chen, V. MOTION CHARACTER AND MOTION WORD RECOGNITION EVALUATION - A motion gesture can be defined in a 3-D space, but hand writing is actually defined on a 2-D surface regardless of the true writing motions… We can consider P^ and V^ as representing the writing motion projected on a vertical plane; VI. USABILITY STUDY, A. Methods - The remote control functions as a mouse, with translation in the vertical plane mapped to the cursor movement on the display with a scale of one meter to 2000 pixels); “upon determining the plane, automatically modify each respective location for each text of the set of text to the determined plane, such that the set of text is aligned with the plane in the extended reality environment and readable by three-dimensional optical character recognition” (Chen, V. MOTION CHARACTER AND MOTION WORD RECOGNITION EVALUATION - A motion gesture can be defined in a 3-D space, but hand writing is actually defined on a 2-D surface regardless of the true writing motions… We can consider P^ and V^ as representing the writing motion projected on a vertical plane); “identifying, by the three-dimensional optical character recognition, one or more characters of each text in the extended reality environment” (Chen, C. Word-Based Motion Word Recognition - In the word-based decoding network, each path is a word model synthesized from corresponding character and ligature HMMs, and the letter sequences are tightly restricted to the vocabulary; D. Letter-Based Motion Word Recognition - We choose the refined HMMs of character and decision-tree clustered ligatures to build the letter-based decoding word network). Although Chen does not explicitly teach, but Gan teaches “generating interactable text based on the identified one or more characters from each text of the set of text; and placing the interactable text on the plane such that the interactable text is a modifiable representation of the set of text” (Gan, 3.2.1. Pre-processing of in-air handwriting trajectories - Considering the fact that characters are defined as the 2- dimensional symbols, the depth information of handwriting trajectories actually is meaningless and useless for representing characters; instead, the x -, y -coordinates should be the most discriminative information for the final recognition. Moreover, even in the 3D space, people always attempt to write texts in an imaginary 2D surface. Therefore, we project each 3D in-air hand- written text into an optimal 2D plane with the minimum mean square error criterion (as shown in Fig. 7) while discarding the depth information. After that, the projected 2D trajectory is further normalized with the slope correction, size normalization, and re-sampling (as shown in Fig. 8). Those normalization steps are important for recognition since handwriting samples suffer great variations in sizes and orientations). Thus, it would have been obvious, in view of Gan to configure Chen’s method as claimed by modifying the projected/recognized text into a “standard” form for the purpose to representing the text in an enhanced visualization.
Claim 8 adds into claim 7 “wherein the set of text comprises one or more words, characters, symbols, or any combination thereof” (Chen, C. Word-Based Motion Word Recognition - In the word-based decoding network, each path is a word model synthesized from corresponding character and ligature HMMs, and the letter sequences are tightly restricted to the vocabulary; D. Letter-Based Motion Word Recognition - We choose the refined HMMs of character and decision-tree clustered ligatures to build the letter-based decoding word network).
Claim 9 adds into claim 7 “receiving one or more commands, wherein the one or more commands are representative of one or more modifications to one or more characteristics of the interactable text; and modifying, based on the one or more commands, the one or more characteristics of the interactable text” (Chen, VI. Usability Study, A. Methods - The time and traverse distance while holding Button B (writing) is recorded separately. The status box displays the recognized word or input status, e.g., writing or recognizing. If an error occurs, the subject re-writes until the word is correctly recognized, and we accumulate the writing time of each trial), “wherein the one or more characteristics comprise size, pitch, yawl, roll, a relative position in the extended virtual reality environment based on the set of coordinates, or any combination thereof” (Chen, Abstract – recognition of characters or words is accomplished based on six-degree-of freedom hand motion data).
Claim 10 adds into claim 7 “determining misspelled text of the set of text; automatically modifying the interactable text associated with misspelled text of the set of text such that the misspelled text is corrected” which Chen suggests in the use of vocabulary to recognize a word combined by separate letters (Chen, V. MOTION CHARACTER AND MOTION WORD RECOGNITION EVALUATION - Given a vocabulary of N words, word-based recognition is formulated as a one-out-of-N problem and becomes more robust to individual letter errors within a word) in which the misspelled word can be recognized and automatically corrected through the stored words in the vocabulary.
As per claim 11, Chen teaches the claimed “method,” comprising: “identifying text from a set of three-dimensional writings in an extended reality environment” (Chen, IV. AIR-WRITING PROCESSING AND MODELING, B. Air-Writing Modeling - HMMs of motion characters are trained directly from the isolated A-to-Z recording, and we create one model for each character… For example, the most complicated letter E has 18 states, and short letters I and J have only eight states. These motion character models are the building blocks for motion words); “determining a plane in the extended reality environment based on a set of coordinates of the text, wherein the set of coordinates comprises one or more coordinates representative of a three-dimensional position of the text in the extended reality environment” (Chen, V. MOTION CHARACTER AND MOTION WORD RECOGNITION EVALUATION - A motion gesture can be defined in a 3-D space, but hand writing is actually defined on a 2-D surface regardless of the true writing motions… We can consider P^ and V^ as representing the writing motion projected on a vertical plane; VI. USABILITY STUDY, A. Methods - The remote control functions as a mouse, with translation in the vertical plane mapped to the cursor movement on the display with a scale of one meter to 2000 pixels); “upon determining the plane, automatically modify each respective coordinate of the set of coordinates for the text, such that the text is aligned with the plane in the extended reality environment” (Chen, V. MOTION CHARACTER AND MOTION WORD RECOGNITION EVALUATION - A motion gesture can be defined in a 3-D space, but hand writing is actually defined on a 2-D surface regardless of the true writing motions… We can consider P^ and V^ as representing the writing motion projected on a vertical plane); “identifying, by three-dimensional optical character recognition, one or more characters of the text in the extended reality environment” (Chen, C. Word-Based Motion Word Recognition - In the word-based decoding network, each path is a word model synthesized from corresponding character and ligature HMMs, and the letter sequences are tightly restricted to the vocabulary; D. Letter-Based Motion Word Recognition - We choose the refined HMMs of character and decision-tree clustered ligatures to build the letter-based decoding word network). Although Chen does not explicitly teach, but Gan teaches “generating interactable text based on the identified one or more characters; and modifying the one or more coordinates such that the interactable text is disposed on the plane (Gan, 3.2.1. Pre-processing of in-air handwriting trajectories - Therefore, we project each 3D in-air hand- written text into an optimal 2D plane with the minimum mean square error criterion (as shown in Fig. 7) while discarding the depth information. After that, the projected 2D trajectory is further normalized with the slope correction, size normalization, and re-sampling (as shown in Fig. 8). Those normalization steps are important for recognition since handwriting samples suffer great variations in sizes and orientations). Thus, it would have been obvious, in view of Gan to configure Chen’s method as claimed by modifying the projected/recognized text into a “standard” form for the purpose to representing the text in an enhanced visualization.
Claim 12 adds into claim 11 “receiving one or more commands from a user, wherein the one or more commands are representative of one or more modifications to one or more characteristics of the interactable text; identifying the one or more modifications from the one or more commands; and modifying, based on the one or more modifications, one or more characteristics of the interactable text” (Chen, VI. Usability Study, A. Methods - The time and traverse distance while holding Button B (writing) is recorded separately. The status box displays the recognized word or input status, e.g., writing or recognizing. If an error occurs, the subject re-writes until the word is correctly recognized, and we accumulate the writing time of each trial; III. AIR-WRITING WITH SIX-DEGREE-OF-FREEDOM MOTION TRACKING - Similar to motion gestures, air-writing is tracked with a continuous stream of sensor data, and the writing is intuitively rendered in the air in unistroke without any pen-up and pen-down information… The system tracks a specially designed handheld device and provides both explicit (position and orientation) and implicit (acceleration and angular speed) 6-DOF data sampled at 60 Hz).
Claim 13 adds into claim 12 “wherein the one or more characteristics include at least one of size, pitch, yawl, roll, and relative position in the extended virtual reality environment based on the set of coordinates” (Chen, Abstract – recognition of characters or words is accomplished based on six-degree-of freedom hand motion data).
Claim 14 adds into claim 13 “wherein the one or more commands include hand movements” (Chen, III. AIR-WRITING WITH SIX-DEGREE-OF-FREEDOM MOTION TRACKING - Air-writing is fundamentally different from conventional handwriting on paper or a surface, which provides no haptic feedback. Similar to motion gestures, air-writing is tracked with a continuous stream of sensor data, and the writing is intuitively rendered in the air in unistroke without any pen-up and pen-down information… The system tracks a specially designed handheld device and provides both explicit (position and orientation) and implicit (acceleration and angular speed) 6-DOF data sampled at 60 Hz).
Claim 15 adds into claim 13 “wherein the one or more commands include hand movements” (Chen, III. AIR-WRITING WITH SIX-DEGREE-OF-FREEDOM MOTION TRACKING - Air-writing is fundamentally different from conventional handwriting on paper or a surface, which provides no haptic feedback. Similar to motion gestures, air-writing is tracked with a continuous stream of sensor data, and the writing is intuitively rendered in the air in unistroke without any pen-up and pen-down information… The system tracks a specially designed handheld device and provides both explicit (position and orientation) and implicit (acceleration and angular speed) 6-DOF data sampled at 60 Hz).
Claim 16 adds into claim 13 “wherein the one or more commands include analog button inputs” (Chen, VI. Usability Study, A. Methods - The time and traverse distance while holding Button B (writing) is recorded separately. The status box displays the recognized word or input status, e.g., writing or recognizing).
Claim 17 adds into claim 13 “wherein the one or more commands include voice commands” which is suggests by Chen (e.g., 1. Introduction - Similarly, in-air handwriting is also suitable for the wearable egocentric cameras (like Google Glass) and it can be treated as an alternative input to speech). It is noted that Chen’s input commands can be generated by any well-known method to control the 3D character recognition method in which the user command is in form of a conventional voice command as a spoken instruction that controls devices, apps, or smart home systems, allowing for hands-free operation by replacing physical interactions like typing or tapping with voice input.
Claim 18 adds into claim 13 “wherein the one or more commands include virtual selections via interactable buttons in the extended reality environment” (Chen, VI. Usability Study, A. Methods - The time and traverse distance while holding Button B (writing) is recorded separately. The status box displays the recognized word or input status, e.g., writing or recognizing).
Claim 19 adds into claim 12 “wherein the one or more commands include at least one of a hand movement, analog button input, voice command, and virtual selection via an interactable button in the extended reality environment” (Chen, VI. Usability Study, A. Methods - The time and traverse distance while holding Button B (writing) is recorded separately. The status box displays the recognized word or input status, e.g., writing or recognizing).
Claim 20 adds into claim 11 “receiving a command and modifying the interactable text based on the command” (Chen, III. AIR-WRITING WITH SIX-DEGREE-OF-FREEDOM MOTION TRACKING - Air-writing is fundamentally different from conventional handwriting on paper or a surface, which provides no haptic feedback. Similar to motion gestures, air-writing is tracked with a continuous stream of sensor data, and the writing is intuitively rendered in the air in unistroke without any pen-up and pen-down information… The system tracks a specially designed handheld device and provides both explicit (position and orientation) and implicit (acceleration and angular speed) 6-DOF data sampled at 60 Hz; VI. Usability Study, A. Methods - The time and traverse distance while holding Button B (writing) is recorded separately. The status box displays the recognized word or input status, e.g., writing or recognizing. If an error occurs, the subject re-writes until the word is correctly recognized, and we accumulate the writing time of each trial).
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 2 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
The term “next to a vertical plane” in claim 2 is a relative term which renders the claim indefinite. The term “next” is not defined by the claim, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention.
Claim 15 is objected to under 37 CFR 1.75 as being a substantial duplicate of claim 14. When two claims in an application are duplicates or else are so close in content that they both cover the same thing, despite a slight difference in wording, it is proper after allowing one claim to object to the other as being a substantial duplicate of the allowed claim. See MPEP § 608.01(m).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PHU K NGUYEN whose telephone number is (571)272-7645. The examiner can normally be reached M-F 8-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Daniel F. Hajnik can be reached at (571) 272-7642. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PHU K NGUYEN/Primary Examiner, Art Unit 2616