DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-5, 8 and 16-19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Bodolec et al. (US 2021/0405738, hereinafter Bodolec), and further in view of Sztuk et al. (US 2020/0209957, hereinafter Sztuk).
Regarding claim 1, Bodolec teaches a wearable device (head mounted device 112, fig. 1 and [0025]: [0026]: HMD 112 is typically worn by user 110 and comprises an electronic display and optical assembly for presenting artificial reality content 122 to user 110) comprising:
a display (display 203, fig. 2A/displays 203A and 203B, fig. 2B; [0047]: HMD 112 includes an interior-facing electronic display 203 configured to present artificial reality content to the user; [0050]: HMD 112 of FIG. 2B includes interior-facing electronic displays 203A and 203B (collectively, “electronic displays 203”) configured to present artificial reality content to the user);
communication circuitry (I/O interfaces 319, fig. 3; [0053]: Processors 302 are further coupled to one or more I/O interfaces 319, which provide I/O interfaces for communicating with other devices such as display devices, image capture devices, other HMDs, and the like. Moreover, the one or more I/O interfaces 319 may include one or more wired or wireless network interface controllers (NICs) for communicating with a network, such as network 104);
a microphone (microphones of HMD 112, [0061]);
a camera (image capture device 138, fig. 1; [0026]: HMD 112 includes one or more sensors (e.g., accelerometers) for tracking motion of the HMD and may include one or more image capture devices 138, e.g., cameras, line scanners and the like, for capturing image data of the surrounding physical environment);
memory (memory 304, fig. 3) comprising one or more storage media storing instructions ([0055]: Memory 304, 314 may comprise any form of memory for storing data and executable software instructions, such as random-access memory (RAM), read only memory (ROM), programmable read only memory (PROM), erasable programmable read only memory (EPROM), electronically erasable programmable read only memory (EEPROM), and flash memory); and
at least one processor (processor(s) 302, fig. 3), wherein the instructions, when executed by at least one processor individually or collectively ([0053]: HMD 112 includes one or more processors 302 and memory 304 that, in some examples, provide a computer platform for executing an operating system 305, which may be an embedded, real-time multitasking operating system, for instance, or other type of operating system. In turn, operating system 305 provides a multitasking operating environment for executing one or more software components 307, including application engine 340; [0113]: Instructions embedded or encoded in a computer-readable storage medium may cause a programmable processor, or other processor, to perform the method, e.g., when the instructions are executed), cause the wearable device to:
control the display to display visual objects (virtual objects 120A and 120B, fig. 1B) respectively corresponding to a plurality of external electronic devices (virtual objects 120A and 120B are related to users 110A and 110B wearing HMD 112A and 112B respectively) identified by using the communication circuitry ([0029]: artificial reality system 10 uses information captured from a real-world, 3D physical environment to render artificial reality content 122 for display to user 110. As one example, artificial reality content 122 may be a consumer gaming application in which user 110 is rendered as an avatar including virtual hand 136 and virtual arm 137 within artificial reality environment 124. Artificial reality content 122 can also include one or more other users rendered as avatars 120 and one or more virtual objects 128A, 128B within the artificial reality environment 124. In some examples, artificial reality content 122 may comprise a mixture of real-world imagery and virtual objects, e.g., mixed reality and/or augmented reality; [0039]: Artificial reality system 20 uses data received from cameras 102, HMDs 112, and controllers 114 to capture 3D information within a real world environment, such as motion by users 110 and/or tracking information with respect to users 110 and objects 108, for use in computing updated pose information for a corresponding frame of reference of HMDs 112. In some examples, each users 110 may be in the same or different real-world environment (e.g., in the same or different physical locations). Either way, the plurality of artificial reality applications may render on the same scene within artificial reality environment 124, based on a current viewing perspective determined for HMD 112C, artificial reality content 122 having virtual objects 128A-128C (collectively, “virtual objects 128”) as spatially overlaid upon real world objects 108A-108C (collectively, “real world objects 108”). Further, from the perspective of HMD 112C, artificial reality system 20 renders avatars 120A, 120B based upon the estimated positions for users 110A, 110B, respectively, within artificial reality environment 124; [0040]: Controllers 114 may be in communication with HMD 112B using near-field communication of short-range wireless communication such as Bluetooth, using wired communication links, or using another type of communication links; [0053]: processors 302 are coupled to electronic display 203, motion sensors 206 and image capture devices 138. Processors 302 are further coupled to one or more I/O interfaces 319, which provide I/O interfaces for communicating with other devices such as display devices, image capture devices, other HMDs, and the like; [0057]: Application engine 320 and rendering engine 322 construct the artificial content for display to user 110 in accordance with current pose information for a frame of reference, typically a viewing perspective of HMD 112, as determined by pose tracker 326. Based on the current viewing perspective, rendering engine 322 constructs the 3D, artificial reality content which may in some cases be overlaid, at least in part, upon the real-world 3D environment of user 110. During this process, pose tracker 326 operates on sensed data received from HMD 112, such as movement information and user commands, and, in some examples, data from any external sensors 90 (FIGS. 1A, 1B), such as external cameras, to capture 3D information within the real world environment, such as motion by user 110 and/or feature tracking information with respect to user 110. Based on the sensed data, pose tracker 326 determines a current pose for the frame of reference of HMD 112 and, in accordance with the current pose, constructs the artificial reality content for communication, via the one or more I/O interfaces 315, to HMD 112 for display to user 110. In some examples, application engine 340 allows a user of HMD 112 to capture image data of the artificial content displayed at electronic device 203; [0097]: In the example illustrated in FIG. 9, artificial reality environment 924 includes avatars 920A, 920B, 920C, 920D, 920E, and 920F (collectively, “avatars 920”), virtual objects 928A and 928B (collectively, “virtual objects 928”), and virtual handheld object 914. In particular, avatars 920 may correspond to other users of HMDs and artificial reality environment 924 may correspond to a gaming application);
identify by using the camera, in a state that the visual objects are displayed, at least one external object comprising at least one hand (identifying a hand by performing object recognition within the image data captured by the image capture devices 138 of HMD; [0032]: artificial reality system 10 performs object recognition within image data captured by image capture devices 138 of HMD 112 to identify hand 132, including optionally identifying individual fingers or the thumb, and/or all or portions of arm 134 of user 110…detecting movement may include tracking positions of one or more of the digits (individual fingers and thumb) of hand 132, including whether one or more digits are placed at an area corresponding to a virtual object (e.g., a UI element). In other examples, detecting movement may include tracking an orientation of hand 132 (e.g., fingers pointing toward HMD 112 or away from HMD 112) and/or an orientation of arm 134 (i.e., the normal of the arm facing toward HMD 112) relative to the current pose of HMD 112, including whether the inside wrist area of hand 132 is facing toward HMD 112. The position and orientation of hand 132 (or a portion thereof) may alternatively be referred to as the pose of hand 132 (or a portion thereof); [0041]: As shown in FIG. 1B, in addition to or alternatively to image data captured via camera 138 of HMD 112C, input data from external cameras 102 may be used to track and detect particular motions, configurations, positions, and/or orientations of hands and arms of users 110, such as hand 132 of user 110C, including movements of individual and/or combinations of digits (fingers, thumb) of the hand; [0048]: HMD 112 may include integrated image capture devices 138A and 138B (collectively, “image capture devices 138”), such as video cameras, laser scanners, Doppler radar scanners, depth scanners, or the like, configured to output image data representative of the physical environment. More specifically, image capture devices 138 capture image data representative of objects (including hand 132) in the physical environment that are within a field of view 130A, 130B of image capture devices 138, which typically corresponds with the viewing perspective of HMD 112; [0058]: gesture detector 324 analyzes objects recognized within image data captured by image capture devices 138 of HMD 112 and/or sensors 90 and external cameras 102 to identify a hand and/or arm of user 110);
identify, based on a direction of the at least one hand (tracking an orientation of hand (e.g., fingers pointing towards HMD 112 or away from HMD 112) is functionally analogous to a direction of the hand), an input (gesture) indicating selection of at least one visual object among the visual objects (avatar 120A or 120B, fig. 1B) displayed in the display (user can select an avatar by pointing at it; [0032]: The artificial reality system 10 analyzes any tracked motions, configurations, positions, and/or orientations of hand 132 and/or portions of arm 134 to identify one or more gestures performed by particular objects, e.g., hand 132 (including particular digits of the hand) and/or portions of arm 134 of user 110. To detect the gesture(s), the artificial reality application may compare the motions, configurations, positions and/or orientations of hand 132 and/or portions of arm 134 to gesture definitions stored in a gesture library of artificial reality system 10, where each gesture in the gesture library may be each mapped to one or more actions; [0033]: User 110 may then select any avatar 120 present at the suspend mode environment to perform additional operations with respect to that user corresponding to the selected avatar, including viewing the user's profile, friend, message, mute, block, or report that user from the suspend mode environment; [0058]: gesture detector 324 analyzes objects recognized within image data captured by image capture devices 138 of HMD 112 and/or sensors 90 and external cameras 102 to identify a hand and/or arm of user 110, and track movements of the hand and/or arm relative to HMD 112 to identify gestures performed by user 110. Gesture detector 324 may track movement, including changes to position and orientation, of arm, hand, and/or digits based on the captured image data, and compare motion vectors of the objects to one or more entries in gesture library 330 to detect a gesture or combination of gestures performed by user 110; [0062]: suspend mode engine 332 allows the user of HMD 112 to select a rendered avatar corresponding to another user from the suspended mode environment and perform one or more operations corresponding to that avatar/user. For example, the user of HMD 112 may select an avatar and view the corresponding user's profile or to friend, message, block, or report that user. In some examples, suspend mode engine 332 may formulate instructions that are to be performed by the artificial reality application server 107 to obtain a profile, friend, message, mute, block, or report a user based on user input on a user interface presented to user 110 at HMD 112; [0101]: In the example shown in FIG. 10, UI element 1054 indicates that the user of HMD 112 may tap a user profile (e.g., any of UI elements 1050) for additional options. In this example, the user has selected avatar 1020A by tapping on UI element 1050C. In some examples, a user of HMD 112 may select an avatar 1020 by looking at the avatar or pointing to them; [0110]: While in suspend mode environment 1124, avatar 1110 may select any of avatars 1120 by performing a selection gesture on any of UI elements 1150A, 1150B, or 1150C (collectively, “UI elements 1150”) to perform additional operations (e.g., as described with reference to UI elements 1050 in FIG. 10). For example, when avatar 1110 selects (e.g., through a touch gesture) one of UI elements 1150); and
based on identification of the input (a gesture selecting an avatar), transmit information associated with the input (sending message or friend request) (HMD 112A or 112B related to the selected user 110A or 110B) corresponding to the at least one visual object (avatar 120A or 120B related to the selected user 110A or 110B) selected by the input (user 110C can select user 110A wearing HMD 112A in the suspend mode by pointing his finger towards user 110A and send a message to avatar 120A of user 110A wearing HMD 112A; [0033]: User 110 may then select any avatar 120 present at the suspend mode environment to perform additional operations with respect to that user corresponding to the selected avatar, including viewing the user's profile, friend, message, mute, block, or report that user from the suspend mode environment. In some examples, artificial reality application server 107 performs artificial reality application backend-network operations to friend, message, block, or report a user from the suspend mode environment. For example, the artificial reality application may formulate instructions that are performed by the artificial reality application server 107 to obtain a profile, friend, message, mute, block, or report a user based on user input on a user interface presented to user 110 at HMD 112; [0062]: suspend mode engine 332 allows the user of HMD 112 to select a rendered avatar corresponding to another user from the suspended mode environment and perform one or more operations corresponding to that avatar/user. For example, the user of HMD 112 may select an avatar and view the corresponding user's profile or to friend, message, block, or report that user. In some examples, suspend mode engine 332 may formulate instructions that are to be performed by the artificial reality application server 107 to obtain a profile, friend, message, mute, block, or report a user based on user input on a user interface presented to user 110 at HMD 112; [0123]: a first user may transmit a message to a second user via an application related to the online social network (e.g., a messaging app)).
Bodolec does not explicitly teach to transmit, based on identification of the input, information associated with the input and an audio signal received through the microphone to at least one external electronic device corresponding to the least one visual object selected by the input.
Sztuk teaches to transmit, based on identification of the input (gaze of the user Ann is used as a gesture input to select Bob from a plurality of other users; monitor detects a gaze 170 from Ann towards the image 136 related to Bob), information associated with the input (parameter such as audio signal level) and an audio signal (audio originated from Ann) received through the microphone (audio established from microphone 194 of Ann’s audio systems 191 is and directed to Bob) to at least one external electronic device (fig. 1 and [0025]: Bob’s wearable display 126 includes an electronic display 122 and an audio system 192) corresponding to the least one visual object (image 136 representing Bob, fig. 1 and [0026] and [0031]) selected by the input (abstract: upon detecting a gaze of the first user at a second user of the AR/VR system or the second user's avatar in the first image, notifying the second user about the gaze of the first user by changing a parameter of an audio communication channel from the first user to the second user, so as to make the first user more audible to the second user; [0008]: The method includes sounding, using the wearable display, an audio to a second user of the AR/VR system, wherein the audio originated from a first user of the AR/VR system and has a parameter of an audio communication channel from the first user to the second user, wherein the parameter is changed from a first value to a second value, so as to make the first user more audible to the second user; [0025]: The audio systems 191,192 may each include at least one of speakers 193 or microphone 194. The images and sounds are conveyed using data received from the controller 152. The controller 152 also establishes a first audio communication channel 171 directed from Ann to Bob; [0026]: the method of FIG. 2 further includes detecting (235) a gaze, e.g. the gaze of the first user (Ann in this case), represented by the line 170, towards a second user (Bob) or the second user's avatar (Bob's avatar 136) in the first image displayed on the first electronic display 121 to the first user (Ann). The detecting 235 may include tracking eye movements of the first user by an eye tracking unit, e.g. the eye tracking system 142. The detecting may include a condition that the gaze duration is longer than a pre-defined threshold amount of time. The detecting may also take into account some eye characteristics like pupil dilation of the first user. The dilated pupils may indicate an elevated level of attention. When the attention monitor 161 determines that one user pays attention to another user for at least the pre-determined threshold amount of time, the other user may be notified. For example, when the monitor 161 detects a gaze 170 from Ann towards the image 136 related to Bob, the controller 152 may notify Bob; [0034]: The method further includes a notification 240. The notification 240 may include changing a parameter of the first audio communication channel 171 from the first user 111 (Ann) to the second user 112 (Bob), so as to make the first user 111 more audible to the second user 112. The parameter is changed from a first, default value, for example a value common for all users, to a second, different value. The parameter may include e.g. the audio signal level, i.e. loudness of Ann's voice when Ann talks to Bob. This may facilitate social interaction between various users of the AR/VR system 100, by making it easier to get attention of Bob, or any other user for that matter; [0035]: Any parameter that makes the voice of the first user more audible or noticeable to the second user may be modified upon gaze detection 235. For example, the parameter may be related to a spectral shape of the audio communication channel. Referring to FIG. 3 with further reference to FIG. 1, the first communication channel 171 (FIG. 1) has a default spectral response function 300 (FIG. 3), which may be common to all users of the AR/VR system 100. The default spectral response function 300 can be characterized by a first peak audio frequency 302 and a first spectral width 304. Upon detecting 235 the gaze of the first user 111 to the second user 112, the controller 152 changes the first communication channel 171 to have a modified spectral response function 310, which is characterized by a second, higher peak audio frequency 312 and a second, narrower, spectral width 314. The modified spectral response function 310 makes the voice sound sharper, and thus more easily noticeable by the second user 112, e.g. when the first user 111 gazes at the second user 112 for a pre-determined amount of time; [0039]: Eventually, the first user 111 may return their attention to the second user 112. When the gaze detection conditions are satisfied, the parameter of the first audio communication channel 171 may be changed to the second value again, or may be changed to a different third value, selected e.g. to make the voice of the first user even more noticeable to the user being looked at; [0040]: The audio has originated from Ann and has a parameter of the audio communication channel 171 from Ann (the first user 111) to Bob (the second user, 112). As explained above, the parameter is changed from the first value to the second value, so as to make Ann (the first user 111) more audible to Bob (the second user 112). This happens when Ann's gaze directed to Bob or Bob's avatar 136 in the image displayed to Ann on Ann's electronic display 121 has been detected by a processor executing instructions to process Ann's eye tracking data. The parameter may be changed from the first value to the second value when the processor executed instructions to determine that eyes of the first user 111 (Ann) have looked at the second user 112 (Bob) or the second user's avatar 136 for at least a predefined time interval). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply Sztuk’s knowledge of transmitting information associated with an input and an audio signal to another user as taught and modify the system of Bodolec because such a system makes the first user more audible to the second user and thereby facilitating a better social interaction between various users of the system by making it easier to get attention ([0034]).
Regarding claim 2, the combination of Bodolec and Sztuk teaches the wearable device of claim 1, wherein the instructions, when executed by at least one processor individually or collectively, cause the wearable device to transmit, to the at least one external electronic device, the information (Sztuk – parameter such as audio signal level) indicating that the audio signal (Sztuk – audio signal originated from Ann and transmitted as part of the first audio communication channel 171) is reproduced (Sztuk – the first communication channel 171 with a modified spectral response function is presented to Bob) at the at least one external electronic device (Sztuk – fig. 1 and [0025]: Bob’s wearable display 126 includes an electronic display 122 and an audio system 192) among the plurality of external electronic devices based on the input (Sztuk - [0034]: The method further includes a notification 240. The notification 240 may include changing a parameter of the first audio communication channel 171 from the first user 111 (Ann) to the second user 112 (Bob), so as to make the first user 111 more audible to the second user 112. The parameter is changed from a first, default value, for example a value common for all users, to a second, different value. The parameter may include e.g. the audio signal level, i.e. loudness of Ann's voice when Ann talks to Bob. This may facilitate social interaction between various users of the AR/VR system 100, by making it easier to get attention of Bob, or any other user for that matter; Sztuk - [0035]: Any parameter that makes the voice of the first user more audible or noticeable to the second user may be modified upon gaze detection 235. For example, the parameter may be related to a spectral shape of the audio communication channel. Referring to FIG. 3 with further reference to FIG. 1, the first communication channel 171 (FIG. 1) has a default spectral response function 300 (FIG. 3), which may be common to all users of the AR/VR system 100. The default spectral response function 300 can be characterized by a first peak audio frequency 302 and a first spectral width 304. Upon detecting 235 the gaze of the first user 111 to the second user 112, the controller 152 changes the first communication channel 171 to have a modified spectral response function 310, which is characterized by a second, higher peak audio frequency 312 and a second, narrower, spectral width 314. The modified spectral response function 310 makes the voice sound sharper, and thus more easily noticeable by the second user 112, e.g. when the first user 111 gazes at the second user 112 for a pre-determined amount of time; Sztuk – [0040]: From the viewpoint of the second user Bob (112), the method includes sounding, using Bob's wearable display, an audio addressed to Bob's attention. The audio has originated from Ann and has a parameter of the audio communication channel 171 from Ann (the first user 111) to Bob (the second user, 112). As explained above, the parameter is changed from the first value to the second value, so as to make Ann (the first user 111) more audible to Bob (the second user 112). This happens when Ann's gaze directed to Bob or Bob's avatar 136 in the image displayed to Ann on Ann's electronic display 121 has been detected by a processor executing instructions to process Ann's eye tracking data. The parameter may be changed from the first value to the second value when the processor executed instructions to determine that eyes of the first user 111 (Ann) have looked at the second user 112 (Bob) or the second user's avatar 136 for at least a predefined time interval).
Regarding claim 3, the combination of Bodolec and Sztuk teaches the wearable device of claim 2, wherein the instructions, when executed by at least one processor individually or collectively, cause the wearable device to transmit (send), to another external electronic device (Bodolec - HMD 112B) being different from the at least one external electronic device (Bodolec - HMD 112B; HMD 112B is different from HMD 112A) among the plurality of external electronic devices, the information indicating cessation (Bodolec – mute; muting a microphone of another user 110B wearing HMD 111C by disabling the microphone of the HMD is functionally analogous to indicating cessation of reproducing of the audio signal) of reproducing of the audio signal by the another external electronic device (Bodolec – [0072]: suspend mode engine 432 implements the functionality to mute another user at HMD 112; Bodolec – [0093]: HMD 112 disables any microphones of HMD 112 or ceases to transmit audio to other users or devices (e.g., other HMDs, console 106) such that the other users in the artificial reality environment cannot hear the user of HMD 112; Sztuk – [0036]: in some embodiments, an active (electronic) outside noise suppression/noise cancellation may be employed, and when a particular user is looked at, the noise cancellation may be turned on or increased for a greater effect. The controller 152 may also be configured to suppress the voices of people located next to a user, especially if these people are talking to each other and not to the user).
Regarding claim 4, the combination of Bodolec and Sztuk teaches the wearable device of claim 2, wherein the instructions, when executed by at least one processor individually or collectively, cause the wearable device to transmit, to the at least one external electronic device (Sztuk – Bob’s wearable display 126, fig. 1), the information indicating to display a visual object (Sztuk - a predefined visual cue such as a color) corresponding to the wearable device (Sztuk - Ann’s wearable display 128) with a preset color associated with the input (Sztuk – [0037]: The notification may also optionally include visual information such as a sign, a visual cue, a text, a symbol, and/or some modification of Ann's avatar 131, such as color, contrast, brightness, outline, etc., added to the Ann's image displayed to Bob on his display 122. The visual cue may represent an artificial image or feature provided by Bob's AR headset. Preferably, the visual cue is predefined and easily noticeable, although the visual cue may remain reasonably subtle. The purpose of the visual cue is to facilitate attracting attention of the second user to the first user, in response to the identified gaze of the first user at or towards the second user).
Regarding claim 5, the combination of Bodolec and Sztuk teaches the wearable device of claim 1, wherein the instructions, when executed by at least one processor individually or collectively, cause the wearable device to transmit, based on a media container (Sztuk - audio communication channel 171) in which the audio signal (Sztuk - audio originated from Ann) received within a time section with a preset period (Sztuk - predetermined time interval) and metadata comprising the information are matched (the audio communication channel comprising the audio and the parameter are transmitted only when the gaze detection system determines that the user has been looking at the other user for a predetermined interval of time; Sztuk – [0032]: For the purpose of identifying a duration of gaze of one user directed to another user of the AR/VR system 100, the attention monitor 161 may calculate a time duration of Ann's gaze at Bob or Bob's avatar 136. The gaze duration may be defined as uninterrupted, continuous gaze at a particular image, such as Bob's avatar 136, or at a particular person or a 3D space location in AR applications, during a predefined time interval. The gaze detection condition may be configured as viewing the particular image for at least a predefined time interval T1. When the gaze detection condition is satisfied, the occurrence of gaze is detected; Sztuk – [0038]: The method may be performed in real time. The AR/VR system 100 can receive updated eye tracking information at regular time intervals as the user's eyes shift between different areas of the screen, and the display system can compute the point of regard, i.e. the gaze point, which is the point in the screen at which the user is looking, based on the updated eye tracking information. Preferably, the point of regard, represented by gaze direction and gaze vergence, is identified at predefined regular time intervals. As described above, the second user is notified as soon as the gaze of the first user is reliably detected, i.e. if the gaze has been occurring for a pre-determined amount of time. Then the system continues monitoring the gaze of the first user 111. The gaze withdrawal condition may be configured as continuing looking at the second user for less than a predefined time interval T2. The two time intervals, T1 (detection of gaze threshold time) and T2 (withdrawal of gaze) may be equal, T1=T2, though not necessarily. The attention monitor 161 may also be configured to detect natural eye and/or head movements of users for the purpose of limiting the duration of the eye contact to a comfortable duration of time. In other words, the first user may briefly look at the floor, turn their head from side to side, etc. The attention monitor 161 may be configured to allow such movements for a “brief withdrawal time” T3 without changing parameters of the audio communication channel between the first and second users. When the gaze withdrawal condition is satisfied, however, the parameter of the first audio communication channel 171 is changed back to the original (first) value), the information and the audio signal (Sztuk - abstract: upon detecting a gaze of the first user at a second user of the AR/VR system or the second user's avatar in the first image, notifying the second user about the gaze of the first user by changing a parameter of an audio communication channel from the first user to the second user, so as to make the first user more audible to the second user; Sztuk - [0008]: The method includes sounding, using the wearable display, an audio to a second user of the AR/VR system, wherein the audio originated from a first user of the AR/VR system and has a parameter of an audio communication channel from the first user to the second user, wherein the parameter is changed from a first value to a second value, so as to make the first user more audible to the second user; Sztuk - [0025]: The audio systems 191,192 may each include at least one of speakers 193 or microphone 194. The images and sounds are conveyed using data received from the controller 152. The controller 152 also establishes a first audio communication channel 171 directed from Ann to Bob; Sztuk - [0026]: the method of FIG. 2 further includes detecting (235) a gaze, e.g. the gaze of the first user (Ann in this case), represented by the line 170, towards a second user (Bob) or the second user's avatar (Bob's avatar 136) in the first image displayed on the first electronic display 121 to the first user (Ann). The detecting 235 may include tracking eye movements of the first user by an eye tracking unit, e.g. the eye tracking system 142. The detecting may include a condition that the gaze duration is longer than a pre-defined threshold amount of time. The detecting may also take into account some eye characteristics like pupil dilation of the first user. The dilated pupils may indicate an elevated level of attention. When the attention monitor 161 determines that one user pays attention to another user for at least the pre-determined threshold amount of time, the other user may be notified. For example, when the monitor 161 detects a gaze 170 from Ann towards the image 136 related to Bob, the controller 152 may notify Bob; Sztuk - [0034]: The method further includes a notification 240. The notification 240 may include changing a parameter of the first audio communication channel 171 from the first user 111 (Ann) to the second user 112 (Bob), so as to make the first user 111 more audible to the second user 112. The parameter is changed from a first, default value, for example a value common for all users, to a second, different value. The parameter may include e.g. the audio signal level, i.e. loudness of Ann's voice when Ann talks to Bob. This may facilitate social interaction between various users of the AR/VR system 100, by making it easier to get attention of Bob, or any other user for that matter; Sztuk - [0035]: Any parameter that makes the voice of the first user more audible or noticeable to the second user may be modified upon gaze detection 235. For example, the parameter may be related to a spectral shape of the audio communication channel. Referring to FIG. 3 with further reference to FIG. 1, the first communication channel 171 (FIG. 1) has a default spectral response function 300 (FIG. 3), which may be common to all users of the AR/VR system 100. The default spectral response function 300 can be characterized by a first peak audio frequency 302 and a first spectral width 304. Upon detecting 235 the gaze of the first user 111 to the second user 112, the controller 152 changes the first communication channel 171 to have a modified spectral response function 310, which is characterized by a second, higher peak audio frequency 312 and a second, narrower, spectral width 314. The modified spectral response function 310 makes the voice sound sharper, and thus more easily noticeable by the second user 112, e.g. when the first user 111 gazes at the second user 112 for a pre-determined amount of time; Sztuk - [0039]: Eventually, the first user 111 may return their attention to the second user 112. When the gaze detection conditions are satisfied, the parameter of the first audio communication channel 171 may be changed to the second value again, or may be changed to a different third value, selected e.g. to make the voice of the first user even more noticeable to the user being looked at; Sztuk - [0040]: The audio has originated from Ann and has a parameter of the audio communication channel 171 from Ann (the first user 111) to Bob (the second user, 112). As explained above, the parameter is changed from the first value to the second value, so as to make Ann (the first user 111) more audible to Bob (the second user 112). This happens when Ann's gaze directed to Bob or Bob's avatar 136 in the image displayed to Ann on Ann's electronic display 121 has been detected by a processor executing instructions to process Ann's eye tracking data. The parameter may be changed from the first value to the second value when the processor executed instructions to determine that eyes of the first user 111 (Ann) have looked at the second user 112 (Bob) or the second user's avatar 136 for at least a predefined time interval).
Regarding claim 8, the combination of Bodolec and Sztuk teaches the wearable device of claim 1, wherein the instructions, when executed by at least one processor individually or collectively, cause the wearable device to, based on whether each of the visual objects is included in an area (Bodolec - area corresponding to a virtual object, [0032]) formed by a location and the direction of the at least one hand within the display (Bodolec - fig. 8 shows the location and direction of hand 136B such that hand 136B is in contact with a virtual object), identify the input indicating selection of the at least one visual object (Bodolec – [0032]: artificial reality system 10 performs object recognition within image data captured by image capture devices 138 of HMD 112 to identify hand 132, including optionally identifying individual fingers or the thumb, and/or all or portions of arm 134 of user 110. Artificial reality system 10 tracks the position, orientation, and configuration of hand 132 (optionally including particular digits of the hand) and/or portions of arm 134 over a sliding window of time … detecting movement may include tracking positions of one or more of the digits (individual fingers and thumb) of hand 132, including whether one or more digits are placed at an area corresponding to a virtual object (e.g., a UI element); Bodolec – [0037]: user 110 may select suspend mode UI element 142 or personal UI element 144 with a selection gesture comprising placement of one or more fingers on or near the corresponding UI element (e.g., on the physical surface on which it rendered or near the physical location corresponding to the UI element); Bodolec – [0040]: HMD 112C may operate substantially similar to HMD 112 of FIG. 1A by tracking hand 132 and/or arm 134 of user 110C. HMD 112A may also operate substantially similar to HMD 112 of FIG. 1A and receive user inputs by tracking movements of hands 132A, 132B of user 110A; Bodolec – [0041]: As shown in FIG. 1B, in addition to or alternatively to image data captured via camera 138 of HMD 112C, input data from external cameras 102 may be used to track and detect particular motions, configurations, positions, and/or orientations of hands and arms of users 110, such as hand 132 of user 110C, including movements of individual and/or combinations of digits (fingers, thumb) of the hand; Bodolec – [0042]: user 110C may select suspend mode UI element 142 with a touch gesture comprising placement of a one or more fingers of the user's hand at an area corresponding to where the suspend mode UI element 142 is rendered such that the one or more virtual fingers of the avatar touches the suspend mode UI element 142 in artificial reality environment 124; Bodolec- [0058]: gesture detector 324 analyzes objects recognized within image data captured by image capture devices 138 of HMD 112 and/or sensors 90 and external cameras 102 to identify a hand and/or arm of user 110, and track movements of the hand and/or arm relative to HMD 112 to identify gestures performed by user 110; Bodolec – [0060]: Other gestures may include a selection gesture comprising placement of one or more fingers of the user's hand near the physical location corresponding to a UI element such that the one or more fingers of a virtual hand of an avatar of the user of HMD 112 comes in contact with the UI element in the artificial reality environment; Bodolec – [0096]: FIG. 8 further shows the user performing a suspend mode selection gesture comprising placement of virtual finger 802 on suspend mode UI element 142, which corresponds to the user placing a finger in the physical area corresponding to where suspend mode UI element is rendered. In response to HMD 112 detecting the selection gesture on suspend mode UI element 142 (e.g. via captured image data of the user's hand), HMD 112 enables the suspend mode feature).
Claims 16, 17, 18 and 19 are similar in scope to claims 1, 2, 3 and 5, respectively, and therefore the examiner provides similar rationale to reject these claims.
Claim(s) 6 and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Bodolec, in view of Sztuk, and further in view of Wu (US 2023/0108331).
Regarding claim 6, the combination of Bodolec and Sztuk does not explicitly teach the wearable device of claim 1, wherein the instructions, when executed by at least one processor individually or collectively, cause the wearable device to identify the input, using a neural network trained to identify the at least one hand which is a preset body part from an image obtained by the camera.
Wu teaches to cause the wearable device ([0089]: A device control method provided in embodiments of this application may be applied to a terminal device such as a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (augmented reality, AR)/virtual reality (virtual reality, VR) device, or a notebook computer) to identify the input, using a neural network trained to identify the at least one hand which is a preset body part (specified body part is a hand) from an image obtained by the camera ([0093]: the image collection device may be a camera; [0096]: the image collection device may first photograph an environment in which the smart television is currently located, collect an environmental image of the user in front of a screen of the smart television, recognize the user in the environmental image by using an NPU of the image collection device, and determine a region in which a human body is located; and then identify a key point (for example, a hand) of the human body, and send a recognition result to a CPU of the image collection device, so that the CPU of the image collection device can check whether the user triggers a preset activation movement or control movement; [0140]: if the specified body part is a hand, the image collection device may recognize the hand in each to-be-recognized region according to a preset algorithm by using a neural network processing unit, and determine a location of the hand of the user based on human body key point information, to recognize a region in which the hand is located in the to-be-recognized region). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply Wu’s knowledge of using a trained neural network to identify a specified body part such as a hand and modify the system of Bodolec and Sztuk because such a system provides fast, real-time detection with enhanced accuracy.
Claim 20 is similar in scope to claim 6, and therefore the examiner provides similar rationale to reject claim 20.
Claim(s) 7, 9-10, and 12-15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Bodolec, in view of Sztuk, and further in view of Nagano et al. (US 2024/0281072, hereinafter Nagano).
Regarding claim 7, the combination of Bodolec and Sztuk teaches the wearable device of claim 1, wherein the instructions, when executed by at least one processor individually or collectively, cause the wearable device to, based on identifying a plurality of hands using the camera (plurality of hands of user 110A as shown in fig. 1B are tracked; Bodolec – [0032]: artificial reality system 10 performs object recognition within image data captured by image capture devices 138 of HMD 112 to identify hand 132, including optionally identifying individual fingers or the thumb, and/or all or portions of arm 134 of user 110. Artificial reality system 10 tracks the position, orientation, and configuration of hand 132 (optionally including particular digits of the hand) and/or portions of arm 134 over a sliding window of time; Bodolec – [0040]: HMD 112C may operate substantially similar to HMD 112 of FIG. 1A by tracking hand 132 and/or arm 134 of user 110C. HMD 112A may also operate substantially similar to HMD 112 of FIG. 1A and receive user inputs by tracking movements of hands 132A, 132B of user 110A; Bodolec – [0041]: As shown in FIG. 1B, in addition to or alternatively to image data captured via camera 138 of HMD 112C, input data from external cameras 102 may be used to track and detect particular motions, configurations, positions, and/or orientations of hands and arms of users 110, such as hand 132 of user 110C, including movements of individual and/or combinations of digits (fingers, thumb) of the hand; Bodolec- [0058]: gesture detector 324 analyzes objects recognized within image data captured by image capture devices 138 of HMD 112 and/or sensors 90 and external cameras 102 to identify a hand and/or arm of user 110, and track movements of the hand and/or arm relative to HMD 112 to identify gestures performed by user 110), a first location and a first direction of a first hand (left hand; as shown in fig. 8, left hand 136A has a first location and a first direction) and a second location and a second direction of a second hand (right hand) being different from the first hand (as shown in fig. 8, right hand 136B has a second location and a second direction which is different from the left hand; Bodolec – [0037]: user 110 may select suspend mode UI element 142 or personal UI element 144 with a selection gesture comprising placement of one or more fingers on or near the corresponding UI element (e.g., on the physical surface on which it rendered or near the physical location corresponding to the UI element); Bodolec – [0060]: Other gestures may include a selection gesture comprising placement of one or more fingers of the user's hand near the physical location corresponding to a UI element such that the one or more fingers of a virtual hand of an avatar of the user of HMD 112 comes in contact with the UI element in the artificial reality environment).
The combination of Bodolec and Sztuk does not explicitly teach to identify the input indicating selection of the at least one visual object based on an area formed by a first location and a first direction of a first hand among the plurality of hands, and a second location and a second direction of a second hand.
Nagano teaches to identify the input indicating selection of the at least one visual object (a grip input operation has been determined based on the virtual object 5 is held in contact with the index finger 38L of the left hand and the index finger 38R of the right hand; [0322]: A method of achieving the grip (hold/pinch) operation with respect to the virtual object 5 can be a method of determining that the virtual object 5 has been gripped in a case where the two fingers are held in contact with the virtual object 5) based on an area (the area occupied by the virtual object 5 that is being held by the index finger 38L of the left hand and the index finger 38R of the right hand as shown in fig. 13) formed by a first location (location of index finger 38L, fig. 13) and a first direction (attitude/orientation/direction of finger pads of index finger 38L, fig. 13) of a first hand (left hand 38L, fig. 13) among the plurality of hands (left hand 38L and right hand 38R, fig. 13), and a second location (location of index finger 38R, fig. 13) and a second direction (attitude/orientation/direction of finger pads of index finger 38R, fig. 13) of a second hand (right hand 38R, fig. 13; [0136]: In the present embodiment, the object recognition unit 27 executes the recognition processing by setting the hands of the users 3a and 3b as targets. Then, the object recognition unit 27 acquires various types of information related to the hands. It should be noted that the hands include fingers; [0137]: The object recognition unit 27 is capable of acquiring, as the recognition result, for example, positions of the right hand and the left hand (including a position relationship between them), attitudes of the right hand and the left hand (including orientations of the hands), motions of the right hand and the left hand (including movement velocity), the presence/absence of an input operation with the right hand and the left hand, and the like; [0140]: the object recognition unit 27 is capable of determining any input operation such as a touch operation, a drag operation, a scroll operation, and a pinch operation; [0142]: the object recognition unit 27 is also capable of acquiring information about directions in which the finger pads are directed, whether each joint of the finger is extended or bent, and by what angle the joint is bent in a case where it is bent with respect to each of the thumb finger, the index finger, the middle finger, the ring finger, and the little finger). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply Nagano’s knowledge of using two hands to select an object as taught and modify the system Bodolec and Sztuk because such a system enables a user to intuitively grip a virtual object in accordance with the size of the virtual object, and thereby enhances the virtual experience of the user ([0330]).
Regarding claim 9, Bodolec teaches a method performed by a wearable device (head mounted device 112, fig. 1 and [0025]: [0026]: HMD 112 is typically worn by user 110 and comprises an electronic display and optical assembly for presenting artificial reality content 122 to user 110), the method comprising:
identifying at least one external electronic devices (virtual objects 120A and 120B are related to users 110A and 110B wearing HMD 112A and 112B respectively) connected through communication circuitry in the wearable device ([0029]: artificial reality system 10 uses information captured from a real-world, 3D physical environment to render artificial reality content 122 for display to user 110. As one example, artificial reality content 122 may be a consumer gaming application in which user 110 is rendered as an avatar including virtual hand 136 and virtual arm 137 within artificial reality environment 124. Artificial reality content 122 can also include one or more other users rendered as avatars 120 and one or more virtual objects 128A, 128B within the artificial reality environment 124. In some examples, artificial reality content 122 may comprise a mixture of real-world imagery and virtual objects, e.g., mixed reality and/or augmented reality; [0039]: Artificial reality system 20 uses data received from cameras 102, HMDs 112, and controllers 114 to capture 3D information within a real world environment, such as motion by users 110 and/or tracking information with respect to users 110 and objects 108, for use in computing updated pose information for a corresponding frame of reference of HMDs 112. In some examples, each users 110 may be in the same or different real-world environment (e.g., in the same or different physical locations). Either way, the plurality of artificial reality applications may render on the same scene within artificial reality environment 124, based on a current viewing perspective determined for HMD 112C, artificial reality content 122 having virtual objects 128A-128C (collectively, “virtual objects 128”) as spatially overlaid upon real world objects 108A-108C (collectively, “real world objects 108”). Further, from the perspective of HMD 112C, artificial reality system 20 renders avatars 120A, 120B based upon the estimated positions for users 110A, 110B, respectively, within artificial reality environment 124; [0040]: Controllers 114 may be in communication with HMD 112B using near-field communication of short-range wireless communication such as Bluetooth, using wired communication links, or using another type of communication links; [0053]: processors 302 are coupled to electronic display 203, motion sensors 206 and image capture devices 138. Processors 302 are further coupled to one or more I/O interfaces 319, which provide I/O interfaces for communicating with other devices such as display devices, image capture devices, other HMDs, and the like; [0057]: Application engine 320 and rendering engine 322 construct the artificial content for display to user 110 in accordance with current pose information for a frame of reference, typically a viewing perspective of HMD 112, as determined by pose tracker 326. Based on the current viewing perspective, rendering engine 322 constructs the 3D, artificial reality content which may in some cases be overlaid, at least in part, upon the real-world 3D environment of user 110. During this process, pose tracker 326 operates on sensed data received from HMD 112, such as movement information and user commands, and, in some examples, data from any external sensors 90 (FIGS. 1A, 1B), such as external cameras, to capture 3D information within the real world environment, such as motion by user 110 and/or feature tracking information with respect to user 110. Based on the sensed data, pose tracker 326 determines a current pose for the frame of reference of HMD 112 and, in accordance with the current pose, constructs the artificial reality content for communication, via the one or more I/O interfaces 315, to HMD 112 for display to user 110);
identifying, based on a direction (tracking an orientation of hand (e.g., fingers pointing towards HMD 112 or away from HMD 112) is functionally analogous to a direction of the hand) in a field-of-view (FOV) of a user wearing the wearable device ([0048]: HMD 112 may include integrated image capture devices 138A and 138B (collectively, “image capture devices 138”), such as video cameras, laser scanners, Doppler radar scanners, depth scanners, or the like, configured to output image data representative of the physical environment. More specifically, image capture devices 138 capture image data representative of objects (including hand 132) in the physical environment that are within a field of view 130A, 130B of image capture devices 138, which typically corresponds with the viewing perspective of HMD 112), an input (gesture) indicating selection of at least one external electronic device (user can select an avatar by pointing at either avatar 120A or 120B, where avatar 120A is related to a user 110A wearing an HMD 112a and avatar 120B is related to a user 110B wearing an HMD 112B, fig. 1B; [0032]: The artificial reality system 10 analyzes any tracked motions, configurations, positions, and/or orientations of hand 132 and/or portions of arm 134 to identify one or more gestures performed by particular objects, e.g., hand 132 (including particular digits of the hand) and/or portions of arm 134 of user 110. To detect the gesture(s), the artificial reality application may compare the motions, configurations, positions and/or orientations of hand 132 and/or portions of arm 134 to gesture definitions stored in a gesture library of artificial reality system 10, where each gesture in the gesture library may be each mapped to one or more actions; [0033]: User 110 may then select any avatar 120 present at the suspend mode environment to perform additional operations with respect to that user corresponding to the selected avatar, including viewing the user's profile, friend, message, mute, block, or report that user from the suspend mode environment; [0034]: user 110 may move the inside wrist area of hand 132 into the current field of view 130 to trigger the display of UI menu 140 (e.g., a virtual icon) near or at least partially over virtual hand 136 and/or virtual arm 137; [0058]: gesture detector 324 analyzes objects recognized within image data captured by image capture devices 138 of HMD 112 and/or sensors 90 and external cameras 102 to identify a hand and/or arm of user 110, and track movements of the hand and/or arm relative to HMD 112 to identify gestures performed by user 110. Gesture detector 324 may track movement, including changes to position and orientation, of arm, hand, and/or digits based on the captured image data, and compare motion vectors of the objects to one or more entries in gesture library 330 to detect a gesture or combination of gestures performed by user 110; [0060]: These predefined gestures to generate UI menu 140 may include moving the inside wrist area (or any area) of the user's hand into the current field of view of HMD 112 (e.g., similar to a user looking at a wrist watch or other wearable device). Other gestures may include a selection gesture comprising placement of one or more fingers of the user's hand near the physical location corresponding to a UI element such that the one or more fingers of a virtual hand of an avatar of the user of HMD 112 comes in contact with the UI element in the artificial reality environment; [0062]: suspend mode engine 332 allows the user of HMD 112 to select a rendered avatar corresponding to another user from the suspended mode environment and perform one or more operations corresponding to that avatar/user. For example, the user of HMD 112 may select an avatar and view the corresponding user's profile or to friend, message, block, or report that user. In some examples, suspend mode engine 332 may formulate instructions that are to be performed by the artificial reality application server 107 to obtain a profile, friend, message, mute, block, or report a user based on user input on a user interface presented to user 110 at HMD 112; [0101]: In the example shown in FIG. 10, UI element 1054 indicates that the user of HMD 112 may tap a user profile (e.g., any of UI elements 1050) for additional options. In this example, the user has selected avatar 1020A by tapping on UI element 1050C. In some examples, a user of HMD 112 may select an avatar 1020 by looking at the avatar or pointing to them; [0110]: While in suspend mode environment 1124, avatar 1110 may select any of avatars 1120 by performing a selection gesture on any of UI elements 1150A, 1150B, or 1150C (collectively, “UI elements 1150”) to perform additional operations (e.g., as described with reference to UI elements 1050 in FIG. 10). For example, when avatar 1110 selects (e.g., through a touch gesture) one of UI elements 1150); and
based on identification of the input (a gesture selecting an avatar), transmitting information associated with the input (sending message or friend request) (HMD 112A or 112B related to the selected user 110A or 110B) through the communication circuitry (user 110C can select user 110A wearing HMD 112A in the suspend mode by pointing his finger towards user 110A and send a message to avatar 120A of user 110A wearing HMD 112A; [0033]: User 110 may then select any avatar 120 present at the suspend mode environment to perform additional operations with respect to that user corresponding to the selected avatar, including viewing the user's profile, friend, message, mute, block, or report that user from the suspend mode environment. In some examples, artificial reality application server 107 performs artificial reality application backend-network operations to friend, message, block, or report a user from the suspend mode environment. For example, the artificial reality application may formulate instructions that are performed by the artificial reality application server 107 to obtain a profile, friend, message, mute, block, or report a user based on user input on a user interface presented to user 110 at HMD 112; [0053]: Processors 302 are further coupled to one or more I/O interfaces 319, which provide I/O interfaces for communicating with other devices such as display devices, image capture devices, other HMDs, and the like; [0062]: suspend mode engine 332 allows the user of HMD 112 to select a rendered avatar corresponding to another user from the suspended mode environment and perform one or more operations corresponding to that avatar/user. For example, the user of HMD 112 may select an avatar and view the corresponding user's profile or to friend, message, block, or report that user. In some examples, suspend mode engine 332 may formulate instructions that are to be performed by the artificial reality application server 107 to obtain a profile, friend, message, mute, block, or report a user based on user input on a user interface presented to user 110 at HMD 112; [0123]: a first user may transmit a message to a second user via an application related to the online social network (e.g., a messaging app)).
Bodolec does not explicitly teach identifying an input indicating a selection based on a direction of a surface of a hand in the field-of-view of a user, and transmitting, based on identification of the input, information associated with the input and an audio signal received through the microphone to at least one external electronic device corresponding to the least one visual object selected by the input.
Sztuk teaches to transmitting, based on identification of the input (gaze of the user Ann is used as a gesture input to select Bob from a plurality of other users; monitor detects a gaze 170 from Ann towards the image 136 related to Bob), information associated with the input (parameter such as audio signal level) and an audio signal (audio originated from Ann) received through the microphone (audio established from microphone 194 of Ann’s audio systems 191 is and directed to Bob) of the wearable device (Ann’s wearable display 128, fig. 1) to at least one external electronic device (fig. 1 and [0025]: Bob’s wearable display 126 includes an electronic display 122 and an audio system 192) through the communication circuitry (abstract: upon detecting a gaze of the first user at a second user of the AR/VR system or the second user's avatar in the first image, notifying the second user about the gaze of the first user by changing a parameter of an audio communication channel from the first user to the second user, so as to make the first user more audible to the second user; [0006]: The wearable displays may each include a visual display module for displaying VR images to the user and/or to augmenting observed outside world with AR imagery and signs, an eye-tracking system for determining such parameters as the user's gaze direction, vergence, eye dilation, etc. in real time, and an audio system for audio communication with other users. The controller may include a remote or a distributed computing system and a communication network connecting the controller to the wearable displays, so that the wearable displays are communicatively coupled with the controller; [0008]: The method includes sounding, using the wearable display, an audio to a second user of the AR/VR system, wherein the audio originated from a first user of the AR/VR system and has a parameter of an audio communication channel from the first user to the second user, wherein the parameter is changed from a first value to a second value, so as to make the first user more audible to the second user; [0025]: With reference to FIG. 1, an AR/VR system 100 enables interaction of two or more users, such as users Ann (111) and Bob (112). The system 100 includes a controller 152 connected via a network with two or more wearable displays 128 (Ann's wearable display) and 126 (Bob's wearable display). Each wearable display 126, 128 includes an electronic display and an audio system, i.e. Ann's wearable display 128 includes an electronic display 121 and an audio system 191, and Bob's wearable display 126 includes an electronic display 122 and an audio system 192. The audio systems 191,192 may each include at least one of speakers 193 or microphone 194. The images and sounds are conveyed using data received from the controller 152. The controller 152 also establishes a first audio communication channel 171 directed from Ann to Bob; [0026]: the method of FIG. 2 further includes detecting (235) a gaze, e.g. the gaze of the first user (Ann in this case), represented by the line 170, towards a second user (Bob) or the second user's avatar (Bob's avatar 136) in the first image displayed on the first electronic display 121 to the first user (Ann). The detecting 235 may include tracking eye movements of the first user by an eye tracking unit, e.g. the eye tracking system 142. The detecting may include a condition that the gaze duration is longer than a pre-defined threshold amount of time. The detecting may also take into account some eye characteristics like pupil dilation of the first user. The dilated pupils may indicate an elevated level of attention. When the attention monitor 161 determines that one user pays attention to another user for at least the pre-determined threshold amount of time, the other user may be notified. For example, when the monitor 161 detects a gaze 170 from Ann towards the image 136 related to Bob, the controller 152 may notify Bob; [0034]: The method further includes a notification 240. The notification 240 may include changing a parameter of the first audio communication channel 171 from the first user 111 (Ann) to the second user 112 (Bob), so as to make the first user 111 more audible to the second user 112. The parameter is changed from a first, default value, for example a value common for all users, to a second, different value. The parameter may include e.g. the audio signal level, i.e. loudness of Ann's voice when Ann talks to Bob. This may facilitate social interaction between various users of the AR/VR system 100, by making it easier to get attention of Bob, or any other user for that matter; [0035]: Any parameter that makes the voice of the first user more audible or noticeable to the second user may be modified upon gaze detection 235. For example, the parameter may be related to a spectral shape of the audio communication channel. Referring to FIG. 3 with further reference to FIG. 1, the first communication channel 171 (FIG. 1) has a default spectral response function 300 (FIG. 3), which may be common to all users of the AR/VR system 100. The default spectral response function 300 can be characterized by a first peak audio frequency 302 and a first spectral width 304. Upon detecting 235 the gaze of the first user 111 to the second user 112, the controller 152 changes the first communication channel 171 to have a modified spectral response function 310, which is characterized by a second, higher peak audio frequency 312 and a second, narrower, spectral width 314. The modified spectral response function 310 makes the voice sound sharper, and thus more easily noticeable by the second user 112, e.g. when the first user 111 gazes at the second user 112 for a pre-determined amount of time; [0039]: Eventually, the first user 111 may return their attention to the second user 112. When the gaze detection conditions are satisfied, the parameter of the first audio communication channel 171 may be changed to the second value again, or may be changed to a different third value, selected e.g. to make the voice of the first user even more noticeable to the user being looked at; [0040]: The audio has originated from Ann and has a parameter of the audio communication channel 171 from Ann (the first user 111) to Bob (the second user, 112). As explained above, the parameter is changed from the first value to the second value, so as to make Ann (the first user 111) more audible to Bob (the second user 112). This happens when Ann's gaze directed to Bob or Bob's avatar 136 in the image displayed to Ann on Ann's electronic display 121 has been detected by a processor executing instructions to process Ann's eye tracking data. The parameter may be changed from the first value to the second value when the processor executed instructions to determine that eyes of the first user 111 (Ann) have looked at the second user 112 (Bob) or the second user's avatar 136 for at least a predefined time interval). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply Sztuk’s knowledge of transmitting information associated with an input and an audio signal to another user as taught and modify the process of Bodolec because such a process makes the first user more audible to the second user and thereby facilitating a better social interaction between various users of the system by making it easier to get attention ([0034]).
Nagano teaches to identifying an input indicating a selection (a grip input operation has been determined based on the virtual object 5 is held in contact with the index finger 38L of the left hand and the index finger 38R of the right hand; [0322]: A method of achieving the grip (hold/pinch) operation with respect to the virtual object 5 can be a method of determining that the virtual object 5 has been gripped in a case where the two fingers are held in contact with the virtual object 5) based on a direction of a surface of a hand (finger pads are a surface of hand, and therefore direction of the finger pads is inherently also the direction of the surface of a hand) in the field-of-view of a user ([0136]: In the present embodiment, the object recognition unit 27 executes the recognition processing by setting the hands of the users 3a and 3b as targets. Then, the object recognition unit 27 acquires various types of information related to the hands. It should be noted that the hands include fingers; [0137]: The object recognition unit 27 is capable of acquiring, as the recognition result, for example, positions of the right hand and the left hand (including a position relationship between them), attitudes of the right hand and the left hand (including orientations of the hands), motions of the right hand and the left hand (including movement velocity), the presence/absence of an input operation with the right hand and the left hand, and the like; [0140]: the object recognition unit 27 is capable of determining any input operation such as a touch operation, a drag operation, a scroll operation, and a pinch operation; [0142]: the object recognition unit 27 is also capable of acquiring information about directions in which the finger pads are directed, whether each joint of the finger is extended or bent, and by what angle the joint is bent in a case where it is bent with respect to each of the thumb finger, the index finger, the middle finger, the ring finger, and the little finger; [0337]: various types of information such as positions, attitudes, and movements of the hand and the respective fingers can be acquired. Moreover, determination of an input operation such as a touch operation, determination of a gesture with the hand, and the like can be executed; [0349]: the user 3 wears an immersive HMD 2 configured to cover the field-of-view of the user 3. Then, the user 3 operates a corresponding virtual object movable in accordance with a movement of the user's hand (fingers) within the VR space by moving the user's hand (fingers)). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply Nagano’s knowledge of determining an input operation based at least on the direction of the finger pads as taught and modify the process of Bodolec and Sztuk because such a process enables a user to intuitively grip a virtual object in accordance with the size of the virtual object, and thereby enhances the virtual experience of the user ([0330]).
Regarding claim 10, the combination of Bodolec, Sztuk and Nagano teaches the method of claim 9, wherein the identifying the input comprises identifying the input based on an area (Bodolec - area corresponding to a virtual object, [0032]) formed based on a direction of the surface of the hand (Bodolec - fig. 8 shows the location and direction of hand 136B such that hand 136B is in contact with a virtual object), and wherein the area comprises a boundary including a location of the hand within the FoV (Bodolec - the area of the surface of virtual object 140 as shown in fig. 8 is the area formed by the virtual object, and the area of the surface of virtual object 142 where the user places the finger is the boundary for selection of the virtual object; Bodolec – [0032]: artificial reality system 10 performs object recognition within image data captured by image capture devices 138 of HMD 112 to identify hand 132, including optionally identifying individual fingers or the thumb, and/or all or portions of arm 134 of user 110. Artificial reality system 10 tracks the position, orientation, and configuration of hand 132 (optionally including particular digits of the hand) and/or portions of arm 134 over a sliding window of time … detecting movement may include tracking positions of one or more of the digits (individual fingers and thumb) of hand 132, including whether one or more digits are placed at an area corresponding to a virtual object (e.g., a UI element); Bodolec – [0037]: user 110 may select suspend mode UI element 142 or personal UI element 144 with a selection gesture comprising placement of one or more fingers on or near the corresponding UI element (e.g., on the physical surface on which it rendered or near the physical location corresponding to the UI element); Bodolec – [0040]: HMD 112C may operate substantially similar to HMD 112 of FIG. 1A by tracking hand 132 and/or arm 134 of user 110C. HMD 112A may also operate substantially similar to HMD 112 of FIG. 1A and receive user inputs by tracking movements of hands 132A, 132B of user 110A; Bodolec – [0041]: As shown in FIG. 1B, in addition to or alternatively to image data captured via camera 138 of HMD 112C, input data from external cameras 102 may be used to track and detect particular motions, configurations, positions, and/or orientations of hands and arms of users 110, such as hand 132 of user 110C, including movements of individual and/or combinations of digits (fingers, thumb) of the hand; Bodolec – [0042]: user 110C may select suspend mode UI element 142 with a touch gesture comprising placement of a one or more fingers of the user's hand at an area corresponding to where the suspend mode UI element 142 is rendered such that the one or more virtual fingers of the avatar touches the suspend mode UI element 142 in artificial reality environment 124; Bodolec - [0048]: HMD 112 may include integrated image capture devices 138A and 138B (collectively, “image capture devices 138”), such as video cameras, laser scanners, Doppler radar scanners, depth scanners, or the like, configured to output image data representative of the physical environment. More specifically, image capture devices 138 capture image data representative of objects (including hand 132) in the physical environment that are within a field of view 130A, 130B of image capture devices 138, which typically corresponds with the viewing perspective of HMD 112; Bodolec- [0058]: gesture detector 324 analyzes objects recognized within image data captured by image capture devices 138 of HMD 112 and/or sensors 90 and external cameras 102 to identify a hand and/or arm of user 110, and track movements of the hand and/or arm relative to HMD 112 to identify gestures performed by user 110; Bodolec – [0060]: Other gestures may include a selection gesture comprising placement of one or more fingers of the user's hand near the physical location corresponding to a UI element such that the one or more fingers of a virtual hand of an avatar of the user of HMD 112 comes in contact with the UI element in the artificial reality environment; Bodolec – [0096]: FIG. 8 further shows the user performing a suspend mode selection gesture comprising placement of virtual finger 802 on suspend mode UI element 142, which corresponds to the user placing a finger in the physical area corresponding to where suspend mode UI element is rendered. In response to HMD 112 detecting the selection gesture on suspend mode UI element 142 (e.g. via captured image data of the user's hand), HMD 112 enables the suspend mode feature).
Regarding claim 12, the combination of Bodolec, Sztuk and Nagano teaches the method of claim 9, wherein the transmitting the information associated with the input and the audio signal received through the microphone of the wearable device, comprises, in a state that a plurality of external electronic devices comprising the at least one external electronic device are connected through the communication circuitry (Bodolec - [0053]: processors 302 are coupled to electronic display 203, motion sensors 206 and image capture devices 138. Processors 302 are further coupled to one or more I/O interfaces 319, which provide I/O interfaces for communicating with other devices such as display devices, image capture devices, other HMDs, and the like; Bodolec – fig. 5 and [0073] shows the peripheral device is in communication with HMD 112, wherein the peripheral device is another wearable device; Sztuk - [0006]: The wearable displays may each include a visual display module for displaying VR images to the user and/or to augmenting observed outside world with AR imagery and signs, an eye-tracking system for determining such parameters as the user's gaze direction, vergence, eye dilation, etc. in real time, and an audio system for audio communication with other users. The controller may include a remote or a distributed computing system and a communication network connecting the controller to the wearable displays, so that the wearable displays are communicatively coupled with the controller), refraining from transmitting the audio signal (Bodolec - blocking a user or disabling the microphone of the user) to another external electronic device different from the at least one external electronic device among the plurality of external electronic devices (the user can block or disable the microphone of the user 110B wearing HMD 112B while continuing to interact with the user 110A wearing HMD 112A; Bodolec – [0043]: HMD 112C will disable any microphones such that avatars 120A, 120B may not hear user 110; Bodolec - [0061]: suspend mode engine 332 disables any microphones of HMD 112 or ceases to transmit audio from HMD 112 to other users such that the other users in the artificial reality environment cannot hear the user of HMD 112; Bodolec – [0065]: blocked users will be unable to see, hear, or otherwise interact with the user and/or his avatar in the artificial reality environment; Bodolec – [0095]: the user may select an avatar of another user to friend, message, mute, block, or report that user from the suspend mode environment (e.g., as described above with reference to FIGS. 1A, 3). In some examples, artificial reality application server 107 performs backend network operations to friend, message, block, or report a user from the suspend mode environment (e.g., as described above with reference to FIGS. 1A, 3). For example, if the user of HMD 112 blocks another user at the network level, the other user will be unable to see or otherwise interact with the user of HMD 112 in the artificial reality environment (or in any other artificial reality environment or application) (and vice versa), to search for the user of HMD 112 on the network, to message the user of HMD 112, or friend the user of HMD 112).
Regarding claim 13, the combination of Bodolec, Sztuk and Nagano teaches the method of claim 9, wherein the transmitting the information associated with the input and the audio signal received through the microphone of the wearable device, comprises transmitting, to the at least one external electronic device (Sztuk – Bob’s wearable display 126, fig. 1), the information causing displaying a preset visual object (Sztuk - a predefined visual cue such as a color) indicating exclusive reproducing of the audio signal (Sztuk – suppressing the voices of other people/noise cancellation is functionally equivalent to exclusively reproducing of the audio signal, i.e. the audio signal is given more prominence while other signals from other users are suppressed; Bodolec – only the selected avatar will reproduce the audio signal from the user while the other avatars are deemphasized; Bodolec – [0110]: avatar 1110 may select avatar 1120A and HMD 112 may render a spotlight or base 1156 under avatar 1120A and present a line 1158 from platform 1160 to base 1056. Base 1056 and line 1058 may move between avatars 1120 as the user selects different avatars 1020. Additionally or alternatively, HMD 112 deemphasizes unselected avatars 1120B and 1120C by, for example, greying or fading them out (represented as dotted lines in FIG. 11); Sztuk – [0036]: In some embodiments, frequency shifting techniques may be applied to change the pitch of the user's voice in real time, to change modeled acoustic parameters of the user's throat in real time to make it sound different, etc. Certain frequencies of the first user's speech may be blocked, and/or background sound may be reduced to emphasize the first user's voice. Furthermore, in some embodiments, an active (electronic) outside noise suppression/noise cancellation may be employed, and when a particular user is looked at, the noise cancellation may be turned on or increased for a greater effect. The controller 152 may also be configured to suppress the voices of people located next to a user, especially if these people are talking to each other and not to the user; Sztuk – [0037]: The notification may also optionally include visual information such as a sign, a visual cue, a text, a symbol, and/or some modification of Ann's avatar 131, such as color, contrast, brightness, outline, etc., added to the Ann's image displayed to Bob on his display 122. The visual cue may represent an artificial image or feature provided by Bob's AR headset. Preferably, the visual cue is predefined and easily noticeable, although the visual cue may remain reasonably subtle. The purpose of the visual cue is to facilitate attracting attention of the second user to the first user, in response to the identified gaze of the first user at or towards the second user).
Regarding claim 14, the combination of Bodolec, Sztuk and Nagano teaches the method of claim 9, comprising, in a state identifying the at least one external electronic device (Bodolec – fig. 1B shows HMD 112 A and 112B associated with users 110A and 110B are identified in the user’s field of view), displaying at least one visual object (Bodolec – avatars 120A and 120B associated with users 110A and 110c are displayed as shown in fig. 1B) representing the at least one external electronic device (Bodolec – HMD 112 A and 112B, fig. 1B) in a display in the wearable device having a display area at least partially overlapped to the FoV (Bodolec – fig. 1B).
Regarding claim 15, the combination of Bodolec, Sztuk and Nagano teaches the method of claim 9, wherein the identifying the input comprises, based on whether the at least one visual object (Bodolec - UI element 142, fig. 8) is included in an area (Bodolec – area where the user’s finger intersect (touch) the surface of UI element 142) extended along the direction of the hand from a location of the hand (Bodolec - fig. 8), identifying the input (Bodolec – [0032]: artificial reality system 10 performs object recognition within image data captured by image capture devices 138 of HMD 112 to identify hand 132, including optionally identifying individual fingers or the thumb, and/or all or portions of arm 134 of user 110. Artificial reality system 10 tracks the position, orientation, and configuration of hand 132 (optionally including particular digits of the hand) and/or portions of arm 134 over a sliding window of time … detecting movement may include tracking positions of one or more of the digits (individual fingers and thumb) of hand 132, including whether one or more digits are placed at an area corresponding to a virtual object (e.g., a UI element); Bodolec – [0037]: user 110 may select suspend mode UI element 142 or personal UI element 144 with a selection gesture comprising placement of one or more fingers on or near the corresponding UI element (e.g., on the physical surface on which it rendered or near the physical location corresponding to the UI element); Bodolec – [0040]: HMD 112C may operate substantially similar to HMD 112 of FIG. 1A by tracking hand 132 and/or arm 134 of user 110C. HMD 112A may also operate substantially similar to HMD 112 of FIG. 1A and receive user inputs by tracking movements of hands 132A, 132B of user 110A; Bodolec – [0041]: As shown in FIG. 1B, in addition to or alternatively to image data captured via camera 138 of HMD 112C, input data from external cameras 102 may be used to track and detect particular motions, configurations, positions, and/or orientations of hands and arms of users 110, such as hand 132 of user 110C, including movements of individual and/or combinations of digits (fingers, thumb) of the hand; Bodolec – [0042]: user 110C may select suspend mode UI element 142 with a touch gesture comprising placement of a one or more fingers of the user's hand at an area corresponding to where the suspend mode UI element 142 is rendered such that the one or more virtual fingers of the avatar touches the suspend mode UI element 142 in artificial reality environment 124; Bodolec- [0058]: gesture detector 324 analyzes objects recognized within image data captured by image capture devices 138 of HMD 112 and/or sensors 90 and external cameras 102 to identify a hand and/or arm of user 110, and track movements of the hand and/or arm relative to HMD 112 to identify gestures performed by user 110; Bodolec – [0060]: Other gestures may include a selection gesture comprising placement of one or more fingers of the user's hand near the physical location corresponding to a UI element such that the one or more fingers of a virtual hand of an avatar of the user of HMD 112 comes in contact with the UI element in the artificial reality environment; Bodolec – [0096]: FIG. 8 further shows the user performing a suspend mode selection gesture comprising placement of virtual finger 802 on suspend mode UI element 142, which corresponds to the user placing a finger in the physical area corresponding to where suspend mode UI element is rendered. In response to HMD 112 detecting the selection gesture on suspend mode UI element 142 (e.g. via captured image data of the user's hand), HMD 112 enables the suspend mode feature).
Claim(s) 11 is/are rejected under 35 U.S.C. 103 as being unpatentable over Bodolec, in view of Sztuk, in view of Nagano, and further in view of Wu (US 2023/0108331).
Regarding claim 11, the combination of Bodolec, Sztuk and Nagano does not explicitly teach the method of claim 9, wherein identifying the input comprises identifying the input using neural network trained to identify the hand which is a preset body part of the user.
Wu teaches identifying the input using a neural network trained to identify the at least one hand which is a preset body part (specified body part is a hand; [0093]: the image collection device may be a camera; [0096]: the image collection device may first photograph an environment in which the smart television is currently located, collect an environmental image of the user in front of a screen of the smart television, recognize the user in the environmental image by using an NPU of the image collection device, and determine a region in which a human body is located; and then identify a key point (for example, a hand) of the human body, and send a recognition result to a CPU of the image collection device, so that the CPU of the image collection device can check whether the user triggers a preset activation movement or control movement; [0140]: if the specified body part is a hand, the image collection device may recognize the hand in each to-be-recognized region according to a preset algorithm by using a neural network processing unit, and determine a location of the hand of the user based on human body key point information, to recognize a region in which the hand is located in the to-be-recognized region). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply Wu’s knowledge of using a trained neural network to identify a specified body part such as a hand and modify the process of Bodolec, Sztuk and Nagano because such a process provides fast, real-time detection with enhanced accuracy.
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Ravasz et al. (US 2020/0387286) describes artificial reality system 10 may identify one gating gesture if a back surface of hand 132 is facing HMD 112, and may identify a different gating gesture if a palm of hand 132 is facing HMD 112. That is, artificial reality system 10 may identify the gating gesture based on certain attributes of hand 132 while hand 132 (or a particular portion thereof) is within the field of view (FoV) of user 110 while user 110 is wearing HMD 112. Further, artificial reality system 10 may identify one gating gesture if the thumb of hand 132 is facing upwards in the FoV of HMD 112, and may identify a different gating gesture if the index finger of hand 132 is facing upwards in the FoV of HMD 112. In some examples, artificial reality system 10 may recognize the particular gestures based on a combination of the various orientation attributes of hand 132 described above.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JWALANT B AMIN whose telephone number is (571)272-2455. The examiner can normally be reached Monday-Friday 10am - 630pm CST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Said Broome can be reached at 571-272-2931. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JWALANT AMIN/Primary Examiner, Art Unit 2612
