DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant's arguments regarding claims 25 – 44, filed 04 December 2025, have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of McCulloch et al. (US 2014/0337023).
The double patenting rejections are maintained.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 25 – 44 rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1, 3, and 5 of U.S. Patent No. 10,580,213. Although the claims at issue are not identical, they are not patentably distinct from each other because claims 1, 3, and 5 of U.S. Patent No. 10,580,213 anticipates claims 25 – 44.
AN: 18/758314
US 10,580,213
a computer-implemented method, comprising: while a computing device is performing a processing operation:
A wearable system for sign language recognition, the wearable system comprising: a head-mounted display configured to present virtual content to a user; an audio sensor configured to detect audio from an environment of the user; an outward-facing imaging system configured to image the environment of the user; and a hardware processor in communication with the head-mounted display and the imaging system, and programmed to:
receiving, by the computing device and from a wearable system, converted text based on sign language detected and translated by the wearable system;
receive a plurality of images captured by the outward-facing imaging system;
detect at least one set of hands in the plurality of images with an object recognizer;
determine a relative size of the at least one set of hands;
identify a source of the at least one set of hands among a plurality of persons based on the relative size of the at least one set of hands;
detect at least one gesture by the at least one set of hands in the plurality of images with the object recognizer;
determine that the source of the at least one gesture belongs to a person other than the user; recognize a meaning of the at least one gesture in a sign language in response to the determination;
identify a target language based on contextual information associated with the user, wherein the contextual information comprises: speech of the user obtained from analysis of audio data obtained by the audio sensor;
translate the at least one gesture into the target language based on the recognized meaning and further based on determining that the source of the at least one gesture belongs to the person other than the user;
generate virtual content based at least partly on a translation of the gesture into the target language; and
rendering, on a graphical user interface of the computing device, the converted text;
cause the head-mounted display to render the virtual content to the user.
retrieving the auxiliary information associated with the converted text; and
displaying, on the computing device, the auxiliary information.
(Claim 3) where in the hardware processor is further programmed to access auxiliary information associated with the gesture; and where the virtual content rendered by the head-mounted display comprises the auxiliary information.
AN: 18/758314
25
35
44
26
36
27
37
28
38
29
39
30
40
31
41
32
42
33
43
34
US 10,580,213
1, 3
1, 3
1, 3
1, 3
1, 3
5
1, 3
1, 3
1, 3
1, 3
Claims 25 – 29, 31 – 39, and 41 - 44 rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1, 6 – 8, 13 – 15, 20 and 21 of U.S. Patent No. 12,055,719. Although the claims at issue are not identical, they are not patentably distinct from each other because claims 1, 6 – 8, 13 – 15, 20 and 21 of U.S. Patent No. 12,055,719 anticipates claims 25 – 29, 31 – 39, and 41 - 44.
AN: 18/758314
US 12,055,719
a computer-implemented method, comprising: while a computing device is performing a processing operation:
A computer-implemented method for sign language recognition, comprising: under control of a wearable system comprising a head-mounted display configured to present virtual content to a user of the wearable system:
receiving, by the computing device and from a wearable system, converted text based on sign language detected and translated by the wearable system;
receiving image information captured by an imaging system of the wearable system in an environment;
detecting, by analyzing the image information and as detected sign language, a presence of sign language in the image information;
identifying a first source of the detected sign language;
determining a change in source to a second source of sign language has occurred;
rendering, on a graphical user interface of the computing device, the converted text;
translating sign language of the second source of sign language into a language understood by the user;
monitoring for a request for auxiliary information associated with the converted text;
monitoring for a request for auxiliary information associated with the language understood by the user;
retrieving the auxiliary information associated with the converted text; and
retrieving the auxiliary information associated with the language understood by the user; and
displaying, on the computing device, the auxiliary information.
displaying, on the wearable system, the auxiliary information.
AN: 18/758314
25
35
44
26
36
27
37
28
38
29
39
30
40
31
41
32
42
33
43
34
US 12,055,719
1
8
15
1
8
15
1
8
15
1
8
15
6
13
20
7
14
21
1
8
15
1
8
15
1
8
15
Claims 30 and 40 rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1, 6 – 8, 13 – 15, 20 and 21 of U.S. Patent No. 12,055,719 in view of Jung et al. (US 2017/0236450).
System of U.S. Patent No. 12,055,719 does not expressly disclose presenting the converted text in an audio format in addition to rendering, on a graphical user interface of the computing device, the converted text. Jung discloses the speech outputter 312 may convert a digital speech generated by the sentence-speech conversion method into an analog speech through digital to audio (D/A) conversion, and output the analog speech to the user (paragraph 83). It would have been obvious to modify the system of U.S. Patent No. 12,055,719 to output audio information of the translated sign language.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 25 – 29, 32 – 39, and 42 – 44 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by McCulloch et al. (US 2014/0337023).
Regarding independent claim 25, McCulloch teaches a computer-implemented method, comprising: while a computing device is performing a processing operation (Figures 4a, 4b):
receiving, by the computing device and from a wearable system (paragraph 35: The optical sensor system 68 of the HMD device 200 may capture image data 80 from the room 308), {1}converted text{1} based on {2}sign language detected and translated{2} by the wearable system (paragraph 51: Using the image data 80, the sign language program 88 may {2}identify the sign language letter or word{2} corresponding to the hand shape formed by the user's right hand 360. The sign language program 88 may convert the letter or word into {1}signed text{1});
rendering, on a graphical user interface of the computing device (paragraph 42: the transparent display 44 of the HMD device 200; paragraph 21: the HMD device 36 may include an eye-tracking system 62 that utilizes at least one inward facing sensor 216. The inward facing sensor 216 may be an image sensor that is configured to acquire image data in the form of eye-tracking data 66 from a user's eyes), the converted text (paragraph 51: The signed text may then be displayed via the transparent display 44 of the HMD device 200);
monitoring for a request for auxiliary information associated with the converted text (paragraph 50: The user 304 may be gazing at the second person 320 as indicated by gaze lines 356. As described above, eye-tracking data 66 corresponding to the user's gaze may be captured by the eye-tracking system 62 and used to determine that the user 304 is focused on face2 56 of the second person 320, or on the second person's right hand 360 that is making a sign language hand shape corresponding to a letter or word);
retrieving the auxiliary information associated with the converted text (paragraph 51: The sign language program 88 may interpret this hand shape and others that form the sentence, "I'm disappointed with the lecture"); and
displaying, on the computing device, the auxiliary information (paragraph 51: This sentence may be displayed as text 40''' in text bubble 362 located above the head of the second person 320. Examiner notes the displayed sentence in the text bubble maps to the rendered converted text and the text bubble itself located above the second person maps to the displayed auxiliary information, which helps associate who the converted text is from).
Regarding dependent claim 26, McCulloch teaches wherein the computing device possesses sign language recognition functionality (paragraph 50: a sign language program 88 may be configured to identify sign language letters and/or words from the image data 80).
Regarding dependent claim 27, McCulloch teaches wherein the processing operation is related to sign language processing (paragraph 50: a sign language program 88 may be configured to identify sign language letters and/or words from the image data 80).
Regarding dependent claim 28, McCulloch teaches wherein the wearable system comprises a head-mounted display configured to present virtual content to a user of the wearable system (paragraph 51 and Figure 3: The signed text may then be displayed via the transparent display 44 of the HMD device 200).
Regarding dependent claim 29, McCulloch teaches wherein: the graphical user interface of the computing device is a component of a head-mounted display (paragraph 42: the transparent display 44 of the HMD device 200; paragraph 21: the HMD device 36 may include an eye-tracking system 62 that utilizes at least one inward facing sensor 216. The inward facing sensor 216 may be an image sensor that is configured to acquire image data in the form of eye-tracking data 66 from a user's eyes); and the converted text rendered on the graphical user interface of the computing device is presented as virtual content overlaid on a physical environment of a user (paragraph 51 and Figure 3: The signed text may then be displayed via the transparent display 44 of the HMD device 200).
Regarding dependent claim 32, McCulloch teaches wherein the wearable system: receives image information captured by an imaging system of the wearable system in an environment (paragraph 35: The optical sensor system 68 of the HMD device 200 may capture image data 80 from the room 308); and detects, by analyzing the image information and as detected sign language, a presence of sign language in the image information (paragraph 51: Using the image data 80, the sign language program 88 may identify the sign language letter or word corresponding to the hand shape formed by the user's right hand 360).
Regarding dependent claim 33, McCulloch teaches wherein the wearable system: identifies a source of the detected sign language (paragraph 50: the second person 320 may be communicating with the third person 324 via a sign language, such as American Sign Language).
Regarding dependent claim 34, McCulloch teaches wherein the wearable system: translates sign language of the source of the detected sign language into a language understood by a user (paragraph 51 and Figure 3: the text translation is outputted to a language for the user).
Regarding claims 35 – 39 and 42 – 44, claims 35 – 39 and 42 – 44 are similar in scope as to claims 25 – 29, 32, and 33, thus the rejections for claims 25 – 29, 32, and 33 hereinabove are applicable to claims 35 – 39 and 42 – 44. McCulloch teaches a non-transitory, computer-readable medium storing one or more instructions executable by a computer system to perform one or more operations (paragraph 71). McCulloch teaches a computer-implemented system, comprising: one or more computers; and one or more computer memory devices (Figures 1 - 2) interoperably coupled with the one or more computers and having tangible, non-transitory, machine-readable media storing one or more instructions that, when executed by the one or more computers, perform one or more operations (paragraphs 13, 71).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 30 and 40 is/are rejected under 35 U.S.C. 103 as being unpatentable over McCulloch et al. (US 2014/0337023) in view of Jung et al. (US 2017/0236450).
Regarding dependent claim 30, McCulloch does not expressly disclose presenting the converted text in an audio format in addition to rendering, on a graphical user interface of the computing device, the converted text. Jung discloses the sentence generator 311 may generate a sentence associated with the sign with respect to the text through a keyword combination corresponding to the text and the used pattern of sign language by the user (paragraph 81) and “The speech outputter 312 may convert the sentence generated by the sentence generator 311 into a speech, and transfer the speech to the user. In a case of keyword-based operation, the speech outputter 312 may apply a sentence-speech conversion method such as a TTS engine to output the speech corresponding to the sentence. The speech outputter 312 may convert a digital speech generated by the sentence-speech conversion method into an analog speech through digital to audio (D/A) conversion, and output the analog speech to the user” (paragraph 83). It would have been obvious for one of ordinary skill in the art at the time of the invention or at the time of the effective filing date of the application to achieve a predictable result of displaying the converted text in addition to outputting an audio translation of the converted text by modifying McCulloch's system that displays the converted text 40''' of a sign language from a person to add audio output of the converted text to the HMD based on the translation of the sign language from a person as taught by Jung, and the result would have been predictable.
Regarding claim 40, claim 40 is similar in scope as to claim 30, thus the rejection for claim 30 hereinabove is applicable to claim 40.
Claims 31 and 41 is/are rejected under 35 U.S.C. 103 as being unpatentable over McCulloch et al. (US 2014/0337023) in view of Forutanpour et al. (US 2014/0081634).
Regarding dependent claim 31, McCulloch does not expressly disclose wherein the request for auxiliary information associated with the converted text is based on a triggering condition, and wherein the triggering condition includes a user indication or detection of a word or phrase that a user may not understand. Forutanpour discloses “keywords within text displayed to the user by the HMD of the AR device may be emphasized. Such emphasis may allow a user to quickly review text for key points being made by the person to whom the text is attributed. Keywords may be determined based on emphasis applied by the person who spoke the word or based on a dictionary that assigns emphasis to particular” (paragraph 61). It would have been obvious for one of ordinary skill in the art at the time of the invention or at the time of the effective filing date of the application to modify McCulloch's system to display keywords with emphasis to attract attention to the user wearing the HMD of important information that may be missed, thus the translation potentially could be misunderstood. One would be motivated to do so because this would allow user to quickly review text for key points (paragraph 61).
Regarding claim 41, claim 41 is similar in scope as to claim 31, thus the rejection for claim 31 hereinabove is applicable to claim 41.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JEFFREY J CHOW whose telephone number is (571)272-8078. The examiner can normally be reached 11AM-7PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Devona Faulk can be reached at 571-272-7515. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JEFFREY J CHOW/Primary Examiner, Art Unit 2618
Prosecution Insights