Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
1. The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
DETAILED ACTION
Claim Rejections - 35 USC § 103
2. Claims 1-3, 8 are rejected under 35 U.S.C. 103 as being unpatentable over Vaught et al. (2013/0187835) in view of Huang (2018/0246698).
As to claim 1, Vaught teaches a system comprising: a handheld electronic device having a microphone configured to detect a voice command ([0046-0047] – computing system 1100 may take the form of a … mobile computing device such as a smart phone or media player, mobile communication device, see-through display system, etc. Computing system 1100 may also optionally include user input devices such as keyboards, mice, game controllers, cameras, microphones, and/or touch screens, for example, as well as the other input devices such as those described above for the see-through display system; and [0020] – the see-through system / head-mounted display system send a user input command to the handheld mobile device selecting the weather tile 208, resulting in the display of weather information on the external display screen 102 of the mobile device 104); and a head-mounted device (head-mounted 100) having a display configured to display an image (Figs. 2-4, 6 and related texts) and the user's intent to interact may be determined via voice recognition analysis of audio input received from the user ([0023-0024] – display system 100 displays menu of items of information that a user browses, via gaze control, voice control or in any other suitable manner; via voice recognition analysis of audio input received from the user), wherein the location of the image is based on a location of the handheld electronic device (Figs. 2-4 and 6 and related texts - referring to FIG. 2, when a user is gazing lines 200, 202 are determined to lead from the eyes of the user to an email user interface tile 204. In response, the see-through display system 100 retrieves and displays a list 206 of messages currently in the inbox of an email account linked to and right next to the handheld electronic user's mobile device 104. Referring to FIG. 3, the user has ceased gazing at the email user interface tile 204, and is instead gazing at a weather user interface tile 208 displayed on the external display screen 102. In response, the see-through display system 100 retrieves and displays a current weather report 210 right next to the handheld electronic user's mobile device 104. Referring to FIG. 4 also shows the display of additional weather-related user interface items 300 on the see-through display. A user may interact with the additional weather-related user interface items 300 via gaze, and selection of one of the additional weather-related user interface items may result in the display of relevant information by the mobile device 104 or right next to the handheld electronic mobile device 104. Referring to FIG. 6, the see-through display system 100 may determine that the user is gazing at the actress by gaze and image analysis, and in response obtain and display information 504 regarding the actress 502. Therefore, Vaught shows or suggests a location of the image is based on a location of the handheld electronic device. When the user gazes at the wall then there is no image displayed next to the handheld electronic user's mobile device 104).
Vaught does not explicitly discuss the microphone in the handheld electronic device receives voice commands.
Huang teaches VR and AR systems typically employ head-worn displays (or helmet-mounted displays, or smart glasses) that are at least loosely coupled to a user's head, and thus move when the end user's head moves. If the end user's head motions are detected by the display system, the data being displayed can be updated to take the change in head pose (i.e., the orientation and/or location of user's head) into account ([0005]); the object selection device 110 may comprise a microphone and corresponding voice interpretation module that, in response to voice commands, can move the 3D cursor 62 over a specific object, and then select the specific object. For example, the end user 50 may speak directional commands, e.g., move left or move right, to incrementally move the 3D cursor 62 over the specific object, and then speak a command, such as “select,” to select the specific object ([0065-0067]); the augmented reality system 100 comprises a local processing and data module 150 operatively coupled, such as by a wired lead or wireless connectivity 152, to components mounted to the head 54 of the end user 50 (e.g., the projection subsystem of the display subsystem 102, microphone assembly 116, speakers 104, and cameras 114, 118). The local processing and data module 150 mounted in a variety of configurations, such as fixedly attached to the frame structure 106 (FIG. 10a), fixedly attached to a helmet or hat 106a (FIG. 10b), embedded in headphones, removably attached to the torso 58 of the end user 50 (FIG. 10c), or removably attached to the hip 59 of the end user 50 in a belt-coupling style configuration ([0087] and Fig. 10d); and the object selection device 110 contained in the local processing and data module 150 ([0090]).
It would have been obvious before the effective filing date of the claimed invention to incorporate the teachings of Huang into the teachings of Vaught for the purpose of having an object selection device that comprise a microphone and corresponding voice interpretation module that, in response to voice commands, can move the 3D cursor over a specific object, and then select the specific object
As to claim 2, Vaught teaches the system defined in claim 1 wherein the handheld electronic device comprises a motion sensor that receives gesture input ([0046-0047] – computing system 1100 may take the form of a … mobile computing device such as a smart phone or media player, mobile communication device, see-through display system, etc. Computing system 1100 may also optionally include user input devices such as keyboards, mice, game controllers, cameras, microphones, and/or touch screens, for example, as well as the other input devices such as those described above for the see-through display system; and [0035] – motion sensor 818 detects movements of a user’s head,… employed as a user input devices, that a user interacts with see through display system via gestures of the eye, neck, head, verbal commands).
As to claim 3, Vaught teaches the system defined in claim 2 wherein the image comprises a virtual control interface manipulated using the gesture input ([0033] - the gaze detection subsystem 810 comprises one or more glint sources 812, such as infrared light sources, configured to cause a glint of light to reflect from each eyeball of a user, and one or more image sensors 814 configured to capture an image of each eyeball of the user. Changes in the glints from the user's eyeballs as determined from image data gathered via image sensor(s) 814 may be used to determine a direction of gaze. Further, a location at which gaze lines projected from the user's eyes intersect the external display may be used to determine an object at which the user is gazing (e.g. a virtual object displayed on an external display). The gaze detection subsystem 810 may have any suitable number and arrangement of light sources and image sensor, [0035] - the motion sensors 818, as well as the microphone(s) 808 and the gaze detection subsystem 810, also may be employed as user input devices, such that a user may interact with see-through display system 800 via gestures of the eye, neck and/or head, as well as via verbal commands).
As to claim 8, Vaught does not explicitly discuss the system defined in claim 1 wherein the handheld electronic device comprises a motion sensor that tracks motion of the handheld electronic device and when the image moves on the display in response to motion of the handheld electronic device. However, Vaught teaches computing system 1100 may take the form of a … mobile computing device such as a smart phone or media player, mobile communication device, see-through display system, etc. Computing system 1100 may also optionally include user input devices such as keyboards, mice, game controllers, cameras, microphones, and/or touch screens, for example, as well as the other input devices such as those described above for the see-through display system ([0046-0047]); and the see-through display system 800 further includes one or more motion sensor 881 to detect movements of a user’s head when the user is wearing the see-through display system 800 ([0035]), hence it would have been obvious that the motion sensor tracks motion of the handheld electronic device since the user is wearing the see-through display system in his/her head and when the user moves the see-through display system also moves and wherein the image moves on the display in response to motion of the handheld electronic device; and when the user move gazes lines are determined to lead from the eyes of the user interface displayed on the mobile device, different images displayed on the handheld electronic device (Figs. 2-4, 6 and related texts).
3. Claims 4-6 are rejected under 35 U.S.C. 103 as being unpatentable over Vaught et al. (2013/0187835) and Huang (2018/0246698) in view of North et al. (2013/0342629).
As to claim 4, Vaught and Huang do not explicitly discuss the system defined in claim 1 wherein the image comprises a live video feed from a video call.
North teaches a live video feed, a still image, a screen capture image from a video or a graphically designed or altered image; the modified image derived from the captured video stream at the devices that is part of the video call ([0059]).
It would have been obvious before the effective filing date of the claimed invention to incorporate the teachings of North into the teachings of Vaught and Huang for the purpose of allowing consumers to communicate with one another using a variety of media including voice, text, and video.
As to claim 5, Vaught teaches the system defined in claim 4 wherein the handheld electronic device comprises a camera ([0047]). North teaches the user interface include an image sensor, an image camera to capture image during video call session ([0040, 0048]).
As to claim 6, North teaches the system defined in claim 5 wherein the display is configured to display the captured images ([0048, 0051]).
4. Claim 7 rejected under 35 U.S.C. 103 as being unpatentable over Vaught et al. (2013/0187835) and Huang (2018/0246698) in view of Otsuka (2023/0139216).
As to claim 7, Vaught and Huang do not explicitly discuss the system defined in claim 1 further comprising control circuitry configured to control whether the image is viewable from an additional head mounter device. However, Vaught teaches sending image information regarding the object to a remote computing device (claim 2).
Otsuka teaches the image processing device causes the flat panel display to display an image in a similar field of view, thereby allowing other people to view the image that the user wearing the head mounted display is viewing ([0026]).
It would have been obvious before the effective filing date of the claimed invention to incorporate the teachings of Otsuka into the teachings of Vaught and Huang for the purpose of allowing other people to view the image that the user wearing the head mounted display is viewing.
6. Claims 9-10 are rejected under 35 U.S.C. 103 as being unpatentable over Vaught et al. (2013/0187835) and Huang (2018/0246698) in view of Prada Gomez et al. (US Patent 8,179,604).
As to claim 9, Vaught and Huang do not explicitly discuss the system defined in claim 1 wherein the handheld electronic device comprises a visual marker with which the head mounted device tracks the location of the handheld electronic device.
Prada Gomez teaches wearable marker comprises an infrared visual marker (abstract; col. 1, line 67 through col. 2, line 1 and lines 65-66); and surface pattern configured to be visible to an IR detection device such as an IR camera or other IR detector (col. 3, lines 5-8); tracking position of the hand-wearable relative to the wearable HMD by measuring position and motion of the particular surface pattern relative to the wearable HMD (col. 1, lines 42-45 and col. 2, lines 8-12).
It would have been obvious before the effective filing date of the claimed invention to incorporate the teachings of Prada Gomez into the teachings of Vaught for the purpose of tracking position and motion of the handheld device and recognize known patterns of motion that correspond to known hand gestures.
As to claim 10, Prada Gomez teaches the system defined in claim 9 wherein the wearable marker comprises an infrared visual marker (abstract; col. 1, line 67 through col. 2, line 1 and lines 65-66) and wherein the head mounted device comprises an infrared camera (col.3, lines 5-8; col. 5, lines 32-35) that detects the infrared marker.
7. Claims 16-17, 20 are rejected under 35 U.S.C. 103 as being unpatentable over Vaught et al. (2013/0187835) in view of Prada Gomez et al. (US Patent 8,179,604) and Balan et al. (2019/0318501).
As to claim 16, Vaught teaches a head-mounted device comprising: a microphone configured to detect a voice command ([0032] – the see-through display system comprises one or more microphone 808 configured to detect sounds such as voice commands from a user); and
a display configured to display a virtual image in response to the voice input ([0020] – the see-through display system (the head-mounted display system) allows a user to select a particular user interface control displayed on the external display screen and an interaction in the context of the user gazing at the weather tile and in response to the user gazing at the weather the head mounted display sends a user input command to the mobile device and selecting the weather tile resulting in the display of weather information; [0023-0024] – the see-through display system displays information about the actress that a user browse, via gaze control, voice control or in any suitable manner; and the user’s intent to interact determined via voice recognition analysis of audio input received from the user with the displayed object determined from sensor input indicating that the user’s gaze is directed at the displayed object), wherein the virtual image is anchored to the handheld electronic device (Figs. 2-4 and 6 and related texts that images and/or information displayed / appeared near or next to the handheld electronic device 104 where the user gazes at, for example, as can be seen in FIG. 2, gaze lines 200, 202 are determined to lead from the eyes of the user to an email user interface tile 204. In response, the see-through display system 100 retrieves and displays a list 206 of messages currently in the inbox of an email account linked to and right next to the handheld electronic the user's mobile device 104); and since the virtual image is anchored to the handheld electronic device, it would have been obvious that the virtual image is controlled based on movement of the handheld electronic device. Vaught does not explicitly discuss an infrared camera configured to track a location of a handheld electronic device.
Prada Gomez teaches surface pattern configured to be visible to an IR detection device such as an IR camera or other IR detector (col. 3, lines 5-8); tracking position of the hand-wearable relative to the wearable HMD by measuring position and motion of the particular surface pattern relative to the wearable HMD (col. 1, lines 42-45 and col. 2, lines 8-12).
Balan teaches the HMD 102 and handheld object 106 are configured to communicate with one another directly, such as via a wireless network connecting utilizing a Bluetooth communication protocol or other suitable wireless communication protocol. In other examples, an optional host computing device 110 may communicate with the HMD 102 and the handheld object 106 to receive data from HMD 102 and handheld objects (including image data acquired by the HMD and IMU data from the handheld objects), to process such data, and to send control signals to these devices ([0022]); the light sources may take any suitable form, such as light-emitting diodes (LEDs) that emit visible light for detection via a visible light camera or cameras on the HMD. Infrared light sources also may be used. In some examples, the camera(s) on the HMD may be configured to filter wavelengths of light other than those emitted by the LEDs to reduce noise levels in the image data. A handheld object may have any suitable number and arrangement of light sources ([0021]); displaying virtual image(s) controlled by the handheld object to appear to wander away from the intended course or direction of movement ([0047]); and the handheld object is body-locked to a location within or on the user's head, the position of the handheld object may change whenever the user's head turns or otherwise moves, regardless of the user's intention to move the handheld object, body-locking the handheld object may comprise locking the pose of the handheld object to the user's neck or elbow. For example, the position of the handheld object may be body-locked to a location on or within the user's neck. When a person moves or turns her head, the person's neck generally does not move or moves just slightly. Accordingly, by body-locking the handheld object to the user's neck, the user's head may turn and move without moving or with only slightly moving a reported position of the handheld object. The body-locked position of the handheld object may still closely track a position of the user's body without undesired updates for head movement alone. For example, the user may walk, lean forward, duck or kneel with the handheld object, and the reported pose of the handheld object may move along with the user ([0074]).
It would have been obvious before the effective filing date of the claimed invention that it is well known to track location of a device with infrared camera and to incorporate the teachings of Prada Gomez and Balan into the teachings of Vaught for the purpose of tracking position and motion of the handheld device and recognize known patterns of motion that correspond to known hand gestures.
As to claim 17, Vaught teaches the head mounted display system sends a user input command to the mobile device selecting the weather tile thereby resulting the display of weather information on the external display of the mobile device, current weather conditions (Figs. 3-4, [0020]).
As to claim 20, Vaught teaches the head mounted device defined in claim 16 wherein a position of the virtual image on the display changes according to changes in the location of the handheld electronic device (Figs. 3-4, 6 and related texts).
8. Claim 18 rejected under 35 U.S.C. 103 as being unpatentable over Vaught, Prada Gomez, and Balan in view of Gharpure et al. (2021/0183379).
As to claim 18, Vaught, Prada Gomez, and Balan do not explicitly teach the head mounted device defined in claim 17 further comprising a speaker configured to play weather sounds in response to the weather related voice input.
Gharpure teaches the head mounted device defined in claim 17 further comprising a speaker configured to play weather sounds in response to the weather related voice input ([0031] - a computer device may include smart speaker, mobile device, wearable device, smart watch functions as a voice command device; [0034] - and physical effect of environment that can perceive by a user or a computing device in include human or animal sounds, e.g., voices or noises; atmospheric sounds, e.g., thunder, rain, wind, or other weather sounds.
It would have been obvious before the effective filing date of the claimed invention to incorporate the teachings of Gharpure into the teachings of Vaught, Prada Gomez, and Balan for the purpose of detecting a current reading location and instruct one or more physical effect devices to enhance the listening experience.
9. Claim 19 is rejected under 35 U.S.C. 103 as being unpatentable over Vaught, Prada Gomez, and Balan in view of Rodriguez, II (2023/0333378).
As to claim 19, Vaught, Prada Gomez, and Balan do not explicitly teach the head mounted device defined in claim 16 wherein the virtual image comprises a virtual control interface for controlling an electronic device and wherein an item on the virtual control interface is selected using gesture input.
Rodriguez teaches the head mounted device wherein the virtual image comprises a virtual control interface for controlling an electronic device and wherein an item on the virtual control interface is selected using gesture input ([0097, 0099], claim 10).
It would have been obvious before the effective filing date of the claimed invention to incorporate the teachings of Rodriguez into the teachings of Vaught, Prada Gomez, and Balan for the purpose of allowing the user controls both the augmented reality eyewear display and the smartwatch display by making hand gestures in the air in close proximity to either the smartwatch or to the graphics displayed by the eyewear.
Response to Arguments
10. Applicant’s arguments with respect to claims 1-10 and 16-20 have been considered but are moot because the new ground(s) of rejection(s).
With respect to claim 1, Applicant argues that Vaught fails to show or suggest the microphone receives voice commands. Examiner respectfully submits please refers to the above rejection(s) with respect to newly cited art Huang (2018/0246698).
With respect to claim 16, Applicant argues that Vaught and Prada Gomez fails to show or suggest virtual image controlled based on movement of the handheld electronic device. Examiner respectfully submits that please refers to the above rejection(s). In response to applicant's arguments against the references individually, one cannot show nonobviousness by attacking references individually where the rejections are based on combinations of references. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981); In re Merck & Co., 800 F.2d 1091, 231 USPQ 375 (Fed. Cir. 1986).
Allowable Subject Matter
11. Claims 11-15 are allowed.
Conclusion
12. Any inquiry concerning this communication or earlier communications from the examiner should be directed to QUYNH H NGUYEN whose telephone number is (571)272-7489. The examiner can normally be reached Monday-Friday 7:30AM-3:30PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Ahmad Matar can be reached on 571-272-7488. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/QUYNH H NGUYEN/Primary Examiner, Art Unit 2693
Prosecution Insights