Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 1/9/2026 has been entered.
Response to Amendment
Applicant's amendments and remarks filed 01/09/2026 have been entered and considered but are not found convincing. Claims 1-2, 6, 13-14, 16-17, 19, 21 have been amended. Claims 22-25 have been added. Claims 4, 9-11, 20 have been cancelled. In summary, claims 1-3, 5-8, 12-19, 21-24 are pending in this application.
Response to Arguments
Claim Rejections under 35 U.S.C. § 112(b)
Applicant has amended claims 13 and 14 to overcome the rejection under 35 U.S.C. § 112(b) as being indefinite. The rejection of claims 13-14 under 35 U.S.C. § 112 (b) has been withdrawn.
Claim Rejections - 35 USC § 103
Applicant's arguments with respect to independent claims have been considered but are moot because the rejection has been modified to address the newly added limitations. The Examiner now relies on the new reference KAMHI.
Claim Objections
Claims 1, 20 are objected to because of the following informalities:.
Claim 1 recites the limitation “determine” in line 5. It should be “determining”.
Claim 20 recites the limitation “The of method of” in line 1. It should be “The method of”
Appropriate correction is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
1. Claims 1, 3, 5-8, 12-16, 18-19 , 21-24 are rejected under 35 U.S.C. 103 as being unpatentable over Abovitz et al. ,IDS, U.S Patent Application Publication No.20150016777(“Abovitz”) in view of Mullins et al, U.S Patent Application Publication No.20160246371 (“Mullins_371”) further in view of KAMHI et al., U.S Patent Application Publication No. 20160180590 (“KAMHI”)
Regarding independent claim 1, Abovitz teaches a method for selectively presenting virtual content to a user in a three-dimensional space (3D) (see at least Figs 61A-61E), the method comprising:
under control of a wearable device comprising a computer processor, a display([0236] Referring to FIG. 15, one embodiment of the head-worn AR system has a suitable user display device (14) as shown in FIG. 15. The user display device may comprise a display lens (82) which may be mounted to a user's head or eyes by a housing or frame (84).”; [0237]”… The depicted system (14) also comprises a head pose processor (36), such as an ASIC (application specific integrated circuit), FPGA (field programmable gate array), and/or ARM processor (advanced reduced-instruction-set machine), which may be configured to calculate real or near-real time user head pose from wide field of view image information output from the capture devices (16). Also shown is another processor (32) configured to execute digital and/or analog processing to derive pose from the gyro, compass, and/or accelerometer data from the sensor assembly (39).”) , and a sensor configured to measure a physiological parameter of a user (see at least [0335] Referring to process flow diagram 3000 of FIG. 30, on a basic level, the AR system is configured to receive input (e.g., visual input 2202 from the user's wearable system, input from room cameras 2204, sensory input 2206 in the form of various sensors in the system, gestures, totems, eye tracking etc.) from one or more AR systems. The AR systems may constitute one or more user wearable systems, and/or stationary room systems (room cameras, etc). The wearable AR systems not only provide images from the FOV cameras, they may also be equipped with various sensors (e.g., accelerometers, temperature sensors, movement sensors, depth sensors, GPS, etc.), as discussed above, to determine the location, and various other attributes of the environment of the user. Of course, this information may further be supplemented with information from stationary cameras mentioned previously that may provide images and/or various cues from a different point of view. It should be appreciated that image data may be reduced to a set of points, as explained above.”; ([0672] In some implementations, the AR system may capture images of the customers, for example via inward facing cameras carried by each customer's individual head worn component. The AR system may provide a composited virtual image to the celebrity of a crowd composed of the various customers.”)):
determine a current location of the user (0017] In another aspect, a method of rendering virtual content to a user is disclosed. The method comprises detecting a location of a user, retrieving a set of data associated with a part of a virtual world model that corresponds to the detected location of the user, wherein the virtual world model comprises data associated with a set of map points of the real world, and rendering, based on the set of retrieved data, virtual content to a user device of the user, such that the virtual content, when viewed by the user, appears to be placed in relation to a set of physical objects in a physical environment of the user.”; [0273] FIG. 21 illustrates an exemplary method 2100 of interacting with the passable world model. First, the user's individual AR system may detect a location of the user (step 2102). The location may be derived by the topological map of the system, as will be described in further detail below. The location may be derived by GPS or any other localization tool.”; [0640] FIG. 63 shows a user sitting in a physical living room space, and using an AR system to experience a virtual room or virtual space in the form of a virtual entertainment or media room, the user executing gestures to interact with a user interface virtual construct to provide input by proxy, according to one illustrated embodiment.”);
obtaining, using the sensor, data associated with the physiological parameter of the user (see at least [0294] More particularly, the user's individual AR system contains information about the user's head pose and orientation in a space, information about hand movement etc. of the user, information about the user's eyes and eye gaze, information about any totems that are being used by the user. Thus, the user's individual AR system already holds a lot of information about the user's interaction within a particular space that is transmitted to the passable world model. This information may then be reliably used to create avatars for the user and help the avatar communicate with other avatars or users of that space. It should be appreciated that no third party cameras are needed to animate the avatar, rather, the avatar is animated based on the user's individual AR system.” ( [0345] With regard to the camera systems, the depicted configuration shows three pairs of cameras: a relative wide field of view ("FOV") or "passive SLAM" pair of cameras arranged to the sides of the user's face, a different pair of cameras oriented in front of the user to handle the Stereo imaging process and also to capture hand gestures and totem/object tracking in front of the user's face. Then there is a pair of Eye Cameras oriented into the eyes of the user so they may attempt to triangulate eye vectors and other information. As noted above, the system may also comprise one or more textured light projectors (such as infrared, or "IR", projectors) to inject texture into a scene.”; [0370] Because the AR system is configured to continuously "know" the physical location and orientation of the user's surroundings, and given that the AR system is constantly collecting various types of data regarding the user's environment (e.g., FOV images, eye tracking data, sensory data, audio data, etc.) conventional types of user inputs may not be necessary. For example, rather than the user physically pressing a button or explicitly speaking a command, user input in the AR system may be automatically recognized. For example, the system may automatically recognize a gesture made by the user's fingers. In another example, the AR system may recognize an input based on eye tracking data. Or, in another example, the AR system may recognize a location, and automatically use that as user input to display virtual content. One important type of user input is gesture recognition in order to perform an action or display virtual content, as will be described below.”);
determining, based at least partly on the data, a physiological state of the user (see at least [0596] To allow user selection of and/or navigation between virtual rooms or virtual spaces, the AR system may be responsive to one or more of, for instance, gestures, voice commands, eye tracking, and/or selection of physical buttons, keys or switches for example carried by a head worn component, belt pack or other physical structure of the individual AR system. The user input may be indicative of a direct selection of a virtual space or room, or may cause a rendering of a menu or submenus to allow user selection of a virtual space or room. [0409] Thus a totem may be any object on which virtual content can be rendered, including for example a body part (e.g., hand) to which virtual content can be locked in a user experience (UX) context. In some implementations, the AR system can render virtual content so as to appear to be coming out from behind a totem, for instance appearing to emerge from behind a user's hand, and slowly wrapping at least partially around the user's hand. The AR system detects user interaction with the virtual content, for instance user finger manipulation with the virtual content which wrapped partially around the user's hand. Alternatively, the AR system may render virtual content so as to appear to emerge from a palm of the user's hand, and detection user fingertip interaction or manipulate of that virtual content. Thus, the virtual content may be locked to a reference from of a user's hand. The AR system may be responsive to various user interactions or gestures, including looking at some item of virtual content, moving hands, touching hands to themselves or to the environment, other gestures, opening and/or closing eyes, etc.”);
evaluating potential virtual content based at least in part on the current location of the user (see at least [0409] Thus a totem may be any object on which virtual content can be rendered, including for example a body part (e.g., hand) to which virtual content can be locked in a user experience (UX) context. In some implementations, the AR system can render virtual content so as to appear to be coming out from behind a totem, for instance appearing to emerge from behind a user's hand, and slowly wrapping at least partially around the user's hand. The AR system detects user interaction with the virtual content, for instance user finger manipulation with the virtual content which wrapped partially around the user's hand. Alternatively, the AR system may render virtual content so as to appear to emerge from a palm of the user's hand, and detection user fingertip interaction or manipulate of that virtual content. Thus, the virtual content may be locked to a reference from of a user's hand. The AR system may be responsive to various user interactions or gestures, including looking at some item of virtual content, moving hands, touching hands to themselves or to the environment, other gestures, opening and/or closing eyes, etc.” [0602] The physical living room may include one or more physical objects, for instance walls, floor, ceiling, a coffee table and sofa. As previously noted, the user may wear a head worn AR system, or head worn component of an AR system, operable to render virtual content in a field of view of the user. For example, the head worn AR system or component may render virtual objects, virtual tools and applications onto the retina of each eye of the user. [0603] As illustrated in FIG. 60A, the user executes a first gesture (illustrated by double headed arrow), to open an icon based cluster user interface virtual construct (FIG. 60B). The gesture may include movement of the user's arms and/or hands or other parts of the user's body, for instance head pose or eyes. Alternatively, the user may use spoken commands to access the icon based cluster user interface virtual construct (FIG. 60B). If a more comprehensive menu is desired, the user may use a different gesture.”; [0654] As illustrated in FIG. 65C, in response the user selection, the AR system may also render a virtual tablet type user interface tool, which provides a more detailed virtual navigation menu than the high level virtual navigation menu. The more detailed virtual navigation menu may include some or all of the menu options of the high level virtual navigation menu, as well as additional options (e.g., retrieve additional content, play interactive game associated with media title or franchise, scene selection, character exploration, actor exploration, commentary). For instance, the AR system may render the detailed virtual navigation menu to, for example, appear to the user as if sitting on a top surface of a table, within arm's reach of the user.”);
selecting first virtual content to be presented to the user from the potential virtual content based at least partly on the evaluation of the current location of the user (see at least [0602] The physical living room may include one or more physical objects, for instance walls, floor, ceiling, a coffee table and sofa. As previously noted, the user may wear a head worn AR system, or head worn component of an AR system, operable to render virtual content in a field of view of the user. For example, the head worn AR system or component may render virtual objects, virtual tools and applications onto the retina of each eye of the user. [0603] As illustrated in FIG. 60A, the user executes a first gesture (illustrated by double headed arrow), to open an icon based cluster user interface virtual construct (FIG. 60B). The gesture may include movement of the user's arms and/or hands or other parts of the user's body, for instance head pose or eyes. Alternatively, the user may use spoken commands to access the icon based cluster user interface virtual construct (FIG. 60B). If a more comprehensive menu is desired, the user may use a different gesture.”; [0654] As illustrated in FIG. 65C, in response the user selection, the AR system may also render a virtual tablet type user interface tool, which provides a more detailed virtual navigation menu than the high level virtual navigation menu. The more detailed virtual navigation menu may include some or all of the menu options of the high level virtual navigation menu, as well as additional options (e.g., retrieve additional content, play interactive game associated with media title or franchise, scene selection, character exploration, actor exploration, commentary). For instance, the AR system may render the detailed virtual navigation menu to, for example, appear to the user as if sitting on a top surface of a table, within arm's reach of the user.”)
determining a spatial location for displaying the virtual content in the 3D space (see at least [0501] Referring now to flowchart 4100 of FIG. 47, in step 4102, the AR system may identify a particular UI. The type of UI may be predetermined by the user. The system may identify that a particular UI needs to be populated based on a user input (e.g, gesture, visual data, audio data, sensory data, direct command, etc.). In step 4104, the AR system may generate data for the virtual UI. For example, data associated with the confines, general structure, shape of the UI etc. may be generated. In addition, the AR system may determine map coordinates of the user's physical location so that the AR system can display the UI in relation to the user's physical location. For example, if the UI is body centric, the AR system may determine the coordinates of the user's physical stance such that a ring UI can be displayed around the user. Or, if the UI is hand centric, the map coordinates of the user's hands may need to be determined. It should be appreciated that these map points may be derived through data received through the FOV cameras, sensory input, or any other type of collected data.”);
generating a virtual user interface comprising at least the determined virtual content; and displaying to the user, via the display of the wearable device, the determined virtual content at the determined spatial location (see at least [0626] As illustrated in FIG. 61E, the AR system renders a comprehensive virtual dashboard menu user interface, for example rendering images to the retina of the user's eyes. The virtual dashboard menu user interface may have a generally annular layout or configuration, at least partially surrounding the user, with various user selectable virtual icons spaced to be within arm's reach of the user.”; [0654] As illustrated in FIG. 65C, in response the user selection, the AR system may also render a virtual tablet type user interface tool, which provides a more detailed virtual navigation menu than the high level virtual navigation menu. The more detailed virtual navigation menu may include some or all of the menu options of the high level virtual navigation menu, as well as additional options (e.g., retrieve additional content, play interactive game associated with media title or franchise, scene selection, character exploration, actor exploration, commentary). For instance, the AR system may render the detailed virtual navigation menu to, for example, appear to the user as if sitting on a top surface of a table, within arm's reach of the user.”) Abovitz is understood to be silent on the remaining limitations of claim 1.
In the same field of endeavor, Mullins_371 teaches a physiological sensor ([0025] In another example embodiment, the device also includes an EEG brain wave sensor coupled to a head of the user to generate brain wave data, a heart rate sensor to generate heart rate data of the user, an eye tracking sensor to determine a gaze of the user on the display, a location sensor to determine a geographic location of the device, and a social network module configured to access social network information associated with the user. The electric wave application identifies the state of mind of the user based on the brain wave data, heart rate data, the gaze of the user, the geographic location of the device, and social network information associated with the user.”;[0037] “The sensors 102 may include electrodes that measure electrical activity from a human. For example, the sensors 102 may include electrodes to measure EEG (electroencephalography) waves of brains, EMG (electromyography) waves of muscles, and EOG (electrooculogram) waves of eyes. The sensors 102 can be used to monitor brainwaves through EEG by detecting electrical signals about a person's level of concentration or state of mind. The sensors may be implemented, for example, by using a headset attached to a head of a user. In another example, the sensors 102 can be used to monitor facial muscles to detect facial expressions of the user”; [0038] In another example embodiment, the sensors 102 may also include: an optical sensor (e.g., a charged-coupled device (CCD)), an orientation sensor (e.g., gyroscope), and/or an audio sensor (e.g., a microphone).) configured to measure a physiological parameter of the user ([0098] In operation 1006, the virtual object generation module 304 identifies a virtual object based on the physical object (or a visual reference on the physical object) and the intensity and pattern of the electric waves of the user. In another example, other parameters may be used to identify the virtual object. For example, heart rate data, voice data may be used to identify the virtual object.”);
determine a current location of the user ([0025] In another example embodiment, the device also includes an EEG brain wave sensor coupled to a head of the user to generate brain wave data, a heart rate sensor to generate heart rate data of the user, an eye tracking sensor to determine a gaze of the user on the display, a location sensor to determine a geographic location of the device, and a social network module configured to access social network information associated with the user. The electric wave application identifies the state of mind of the user based on the brain wave data, heart rate data, the gaze of the user, the geographic location of the device, and social network information associated with the user.”) :
obtaining, using the physiological sensor, data associated with the physiological parameter of the user ([0025] In another example embodiment, the device also includes an EEG brain wave sensor coupled to a head of the user to generate brain wave data, a heart rate sensor to generate heart rate data of the user, an eye tracking sensor to determine a gaze of the user on the display, a location sensor to determine a geographic location of the device, and a social network module configured to access social network information associated with the user. The electric wave application identifies the state of mind of the user based on the brain wave data, heart rate data, the gaze of the user, the geographic location of the device, and social network information associated with the user.”);
determining, based at least partly on the data ([0098] In operation 1006, the virtual object generation module 304 identifies a virtual object based on the physical object (or a visual reference on the physical object) and the intensity and pattern of the electric waves of the user. In another example, other parameters may be used to identify the virtual object. For example, heart rate data, voice data may be used to identify the virtual object.”), a physiological state of the user(see at least at [0025] of Mullins “In another example embodiment, the device also includes an EEG brain wave sensor coupled to a head of the user to generate brain wave data, a heart rate sensor to generate heart rate data of the user, an eye tracking sensor to determine a gaze of the user on the display, a location sensor to determine a geographic location of the device, and a social network module configured to access social network information associated with the user. The electric wave application identifies the state of mind of the user based on the brain wave data, heart rate data, the gaze of the user, the geographic location of the device, and social network information associated with the user.[0030] Biometrics data from the user may also be used to program a visual gesture resulting in a command or user input in the HMD. For example, the visual gesture may include detecting that a user of the HMD is staring at a physical object for more than a time duration threshold in combination with a high heart rate. Therefore, when a user's eye gaze intently directed towards an object (e.g., a switch blinking red) for more than 3 seconds, and the user's heart beat exceeds a threshold heartbeat, the HMD generates a specific AR content (e.g., instructions or operation of the switch, virtual arrows telling the user to act on the switch) without the user having to use his hands to tap on a touch sensitive surface on the HMD.”; [0053] The state of mind identification module 210 may thus determine and identify a state of mind of the user based on the intensity module 204 and the pattern module 206. For example, the state of mind identification module 210 may determine that the user is happy, relaxed, angry, focused, hungry, or thirsty. In another embodiment, the state of mind identification module 210 may determine a specific object (e.g., a minivan) or a specific action (e.g., open a door) that the user is thinking.);
evaluating potential virtual content based at least in part on the physiological state of the user ([0045] The storage device 108 may be configured to store a database of visual references, virtual objects corresponding to the visual references, features of the virtual objects corresponding to the virtual objects, and corresponding states of mind. The features of the virtual objects can change with the state of mind of the user. For example, the color of the virtual chair can change from blue to red as the user becomes more focused. The virtual chair may be displayed in a blue color if the user is relaxed. In another example, features of the virtual object change when the features are present in the focus area of the display 104. For example, the visual reference may include a machine-readable code or a previously identified image (e.g., a picture of shoe). The previously identified image of the show may correspond to a three-dimensional virtual shoe that can be viewed from different angles by manipulating the position of the device 100 relative to the picture of the shoe. Features of the three-dimensional virtual shoe may include selectable icons on the three-dimensional virtual shoe. An icon may be selected or activated by moving (e.g., repositioning, reorienting, or both) the device 100 to display the icon within a focus area of the display 104. For example, the focus area may be a central area of the display 104, a corner of the display 104, an edge of the display 104, or any suitable combination thereof. [0058] The virtual object generation module 304 generates and displays a visualization of a three-dimensional virtual object engaged with an image of the physical object captured by the sensor 102 of the device 100 (e.g., the virtual sofa floats and rotates on top of the magazine page). The virtual object may be based on the visual reference (e.g., a furniture ad in the magazine page) and a state of mind of the user. In one embodiment, each virtual object may be uniquely associated with a visual reference and a particular state of mind. The virtual object generation module 304 renders the visualization of the virtual object based a position of the device 100 relative to the visual reference. In another embodiment, attributes of the virtual object may be based on the state of mind of the user. For example, the virtual object generation module 304 may generate a blue color sofa when the state of mind of the user indicates that the user is relaxed and is thinking of a sofa. Similarly, the virtual object generation module 304 may generate a red color sofa when the state of mind of the user indicates that the user is excited and thinking of a sofa. [0079] The device 600 generates a visualization of a three-dimensional virtual object in a display 602 of the device 600 based on outputs from sensors 610 and the visual reference 606. For example, the device 600 may determine that the user 601 is geographically located at an architectural firm. The device 600 determines from the sensors 610 that the state of mind of the user 601 is focused on a high rise building. In another embodiment, a front facing camera 614 of the device 600 may further enhance and provide additional data on the state of mind of the user 601. For example, the device 600 may obtain a live picture of the user 601 using the front facing camera 614 to determine a smile or a frown. In another example, the front facing camera 614 may be used for facial recognition to determine the identity of the user 601. The device 600 may retrieve preferences from the user 601 such as, for example, favorite colors or items. In another example, the device 600 determines, identifies, and manipulates a virtual object to be displayed in the display 602 based on a combination of the geographic location of the device 600 (e.g., office, home, restaurant, city, country), time of capture (e.g., morning, afternoon, evening, holiday, weekend) of the visual reference 606, orientation (e.g., portrait or landscape, how close) of the device 600 relative to the visual reference 606, identification of the user 601 (e.g. using facial recognition, or login information), preferences of the user 601 (e.g., favorite color, favorite type of music) social network information (e.g, number of friends, interests, proximity of friends, postings) related to the user 601, outputs from sensors 610 (e.g., EEG brain waves, EMG muscles waves, EOG eyes waves, heart rate, blood pressure), and the visual reference 606..);
selecting first virtual content to be presented to the user from the potential virtual content based at least partly on the evaluation of the physiological state of the user (see at least [0045] The storage device 108 may be configured to store a database of visual references, virtual objects corresponding to the visual references, features of the virtual objects corresponding to the virtual objects, and corresponding states of mind. The features of the virtual objects can change with the state of mind of the user. For example, the color of the virtual chair can change from blue to red as the user becomes more focused. The virtual chair may be displayed in a blue color if the user is relaxed. In another example, features of the virtual object change when the features are present in the focus area of the display 104. For example, the visual reference may include a machine-readable code or a previously identified image (e.g., a picture of shoe). The previously identified image of the show may correspond to a three-dimensional virtual shoe that can be viewed from different angles by manipulating the position of the device 100 relative to the picture of the shoe. Features of the three-dimensional virtual shoe may include selectable icons on the three-dimensional virtual shoe. An icon may be selected or activated by moving (e.g., repositioning, reorienting, or both) the device 100 to display the icon within a focus area of the display 104. For example, the focus area may be a central area of the display 104, a corner of the display 104, an edge of the display 104, or any suitable combination thereof. [0079] The device 600 generates a visualization of a three-dimensional virtual object in a display 602 of the device 600 based on outputs from sensors 610 and the visual reference 606. For example, the device 600 may determine that the user 601 is geographically located at an architectural firm. The device 600 determines from the sensors 610 that the state of mind of the user 601 is focused on a high rise building. In another embodiment, a front facing camera 614 of the device 600 may further enhance and provide additional data on the state of mind of the user 601. For example, the device 600 may obtain a live picture of the user 601 using the front facing camera 614 to determine a smile or a frown. In another example, the front facing camera 614 may be used for facial recognition to determine the identity of the user 601. The device 600 may retrieve preferences from the user 601 such as, for example, favorite colors or items. In another example, the device 600 determines, identifies, and manipulates a virtual object to be displayed in the display 602 based on a combination of the geographic location of the device 600 (e.g., office, home, restaurant, city, country), time of capture (e.g., morning, afternoon, evening, holiday, weekend) of the visual reference 606, orientation (e.g., portrait or landscape, how close) of the device 600 relative to the visual reference 606, identification of the user 601 (e.g. using facial recognition, or login information), preferences of the user 601 (e.g., favorite color, favorite type of music) social network information (e.g, number of friends, interests, proximity of friends, postings) related to the user 601, outputs from sensors 610 (e.g., EEG brain waves, EMG muscles waves, EOG eyes waves, heart rate, blood pressure), and the visual reference 606..0081] FIG. 6B is a block diagram illustrating an example of a visualization of an action on a virtual object in the device based on thought, according to some example embodiments. The device 600 determines a change in the state of mind of the user 601 (e.g., the user is thinking about rain on the building). The device 600 then generates a change in the visualization of the three-dimensional virtual object in the display 602 of the device 600 based on the change in the state of mind of the user 601 in response to changes in outputs from sensors 610 and the front facing camera 614. For example, rain over the building 608 may be dynamically animated in the display 602 when the device 600 detects that the user 601 has focused on rain. [0082] As such, changes of the already displayed three-dimensional virtual object in the display 602 are determined based on the changes in the state of mind of the user 601. In another example, the color of the building 608 may change to a lighter hue when the user 601 becomes more relaxed while looking at the building 608. In another example, the texture of the building 608 may change to a rougher texture when the user 601 becomes agitated.”);
determining a location for displaying the virtual content in the 3D space (see at least at [0043] The augmented reality application 112 may generate and display a visualization of the virtual object engaged with an image or picture of the physical object in the display 104. The virtual object may be generated based on the visual reference and the state of mind of the user. Each virtual object may correspond to a unique visual reference and corresponding state of mind (e.g., unique to that virtual object within the augmented reality application 112). In another embodiment, the augmented reality application 112 renders the visualization of the virtual object based a position and an orientation of the device 100 relative to the visual reference in the physical object.; [0079] The device 600 generates a visualization of a three-dimensional virtual object in a display 602 of the device 600 based on outputs from sensors 610 and the visual reference 606. For example, the device 600 may determine that the user 601 is geographically located at an architectural firm. The device 600 determines from the sensors 610 that the state of mind of the user 601 is focused on a high rise building. In another embodiment, a front facing camera 614 of the device 600 may further enhance and provide additional data on the state of mind of the user 601”).;
generating a virtual user interface comprising at least the determined virtual content (see at least at [0076] The action module 504 may be configured to trigger an action similar to tapping on the icon on the display 104. For example, the action module 504 may generate a message notification, a dialog box, a menu, or any other action triggered by the presence of the feature in the focus area in the display 104 or by the state of mind of the user. In another embodiment, the action module 404 may be configured to generate a communication from the device 100 to another device, for example, via a wireless network.”; 0092] In another example, the device 600 may display a virtual menu of TV channels overlaid on the TV based on the state of mind of the user 601. For example, if the user is excited, the menu of TV channels may include sports channels and action movies. In another example, the user 601 may look through a transparent display of the device 600 to a radio device. Similarly, a virtual menu of music channels may be displayed over the radio device based on the state of mind of the user 601. For example, the device 600 may display a virtual menu of classical or relaxing music channels when sensors 610 indicate that the user 601 is relaxed or sleepy.”) and
display to the user, via the display of the wearable device, the determined virtual content at the determined location (see at least at [0081] FIG. 6B is a block diagram illustrating an example of a visualization of an action on a virtual object in the device based on thought, according to some example embodiments. The device 600 determines a change in the state of mind of the user 601 (e.g., the user is thinking about rain on the building). The device 600 then generates a change in the visualization of the three-dimensional virtual object in the display 602 of the device 600 based on the change in the state of mind of the user 601 in response to changes in outputs from sensors 610 and the front facing camera 614. For example, rain over the building 608 may be dynamically animated in the display 602 when the device 600 detects that the user 601 has focused on rain.[0082] As such, changes of the already displayed three-dimensional virtual object in the display 602 are determined based on the changes in the state of mind of the user 601. In another example, the color of the building 608 may change to a lighter hue when the user 601 becomes more relaxed while looking at the building 608. In another example, the texture of the building 608 may change to a rougher texture when the user 601 becomes agitated.)
Therefore, it would have been obvious to one of ordinary skill in art before the effective filling date to the claimed invention to modify display virtual content associated with physical surface of Abovitz with inducing sensor data such as the brain wave data, heart rate data, the gaze of the user as seen in Mullins_371 because this medication would identify the state of mind of the user ([0025 of Mullins_371) Both Abvitz and Mullins_371 are understood to be silent on the remaining limitations of claim 1.
In the same field of endeavor, KAMHI teaches evaluating potential virtual content based at least in part on the physiological state of the user and the current location of the user ([0022] FIG. 1 graphically represents an example of augmented reality according to one embodiment. In this example, a user 110 starts an AR module (not shown) on a handheld device 112 (e.g., computer tablet) and is presented with a suggested list of virtual 3D models. The list may be based on the user's location, mood, and/or user profile. For example, if the AR module determines that the user 110 is located in corn field that appears as a “sea” of corn stalks 114, and that the user 110 is in a playful mood, the list may include a virtual 3D model comprising a “flying whale” configured to playfully dive in and out of the corn stalks 114. After selecting the virtual 3D model from the list, the user 110 captures a micro video clip by positioning a world facing 3D camera of the handheld device 112 (e.g., tablet) to a view of interest (e.g., an area in and above the corn field). The AR module displays a virtual image 116 of the flying whale on a screen of the handheld device 112. The virtual image 116 of the flying whale blends into the user's environment and flies in the 3D camera's field of view as if part of the user's real environment. Those skilled in the art will recognize from the disclosure herein that unlike the user 110 shown in FIG. 1, the whale image 118 does not actually exist in the surrounding environment. Rather, the whale image 118 is shown in FIG. 1 to represent the augmented reality experienced by the user 110 when viewing a video of the virtual image 116 of the flying whale through the screen on the handheld device 112.” [0039] In certain embodiments, the method 800 does not require any training stage or binding of a virtual 3D model to the detection and tracking of a 2D or 3D object. The type of potential augmentation may be suggested to the user based on the user's location, mood, and/or profile. Once the type of augmentation is chosen, the rendering of the augmentation into the reality is driven by the people and the objects in the scene (e.g., within the field of view of the 3D video capture device). The augmented virtual 3D objects interact with the physical objects in the scene in a natural way. For example, if the person waves his hands, a virtual birds follows his hands, a virtual monkey climbs on to the shoulder of a person and interacts with the person as it would in real life.);
selecting first virtual content to be presented to the user from the potential virtual content based at least partly on the evaluation of the physiological state of the user and the current location of the user ([0022] FIG. 1 graphically represents an example of augmented reality according to one embodiment. In this example, a user 110 starts an AR module (not shown) on a handheld device 112 (e.g., computer tablet) and is presented with a suggested list of virtual 3D models. The list may be based on the user's location, mood, and/or user profile. For example, if the AR module determines that the user 110 is located in corn field that appears as a “sea” of corn stalks 114, and that the user 110 is in a playful mood, the list may include a virtual 3D model comprising a “flying whale” configured to playfully dive in and out of the corn stalks 114. After selecting the virtual 3D model from the list, the user 110 captures a micro video clip by positioning a world facing 3D camera of the handheld device 112 (e.g., tablet) to a view of interest (e.g., an area in and above the corn field). The AR module displays a virtual image 116 of the flying whale on a screen of the handheld device 112. The virtual image 116 of the flying whale blends into the user's environment and flies in the 3D camera's field of view as if part of the user's real environment. Those skilled in the art will recognize from the disclosure herein that unlike the user 110 shown in FIG. 1, the whale image 118 does not actually exist in the surrounding environment. Rather, the whale image 118 is shown in FIG. 1 to represent the augmented reality experienced by the user 110 when viewing a video of the virtual image 116 of the flying whale through the screen on the handheld device 112.”)
Therefore, it would have been obvious to one of ordinary skill in art before the effective filling date to the claimed invention to modify display virtual content associated with physical surface of Abovitz and inducing sensor data such as the brain wave data, heart rate data, the gaze of the user of Mullins_371 with evaluating potential virtual content based on user's location and mood of KAMHI because this modification would automatically selects and/or recommends a virtual 3D model based on the context data ([0036] of KAMHI)
Thus, the combination of Abovitz, Mullins_371 and KAMHI teaches a method for selectively presenting virtual content to a user in a three-dimensional space (3D), the method comprising: under control of a wearable device comprising a computer processor, a display, and a physiological sensor configured to measure a physiological parameter of a user: determine a current location of the user; obtaining, using the physiological sensor, data associated with the physiological parameter of the user; determining, based at least partly on the data, a physiological state of the user; evaluating potential virtual content based at least in part on the physiological state of the user and the current location of the user; selecting first virtual content to be presented to the user from the potential virtual content based at least partly on the evaluation of the physiological state of the user and the current location of the user; determining a spatial location for displaying the virtual content in the 3D space; generating a virtual user interface comprising at least the determined virtual content; and displaying to the user, via the display of the wearable device, the determined virtual content at the determined spatial location.
Regarding claim 3, Abovitz, Mullins_371 and KAMHI teach the method of claim 1, further comprising obtaining the data associated with the physiological parameter of the user using an inward-facing imaging system configured to image one or both eyes of the user (see at least [0672] of Abovitz “ In some implementations, the AR system may capture images of the customers, for example via inward facing cameras carried by each customer's individual head worn component. The AR system may provide a composited virtual image to the celebrity of a crowd composed of the various customers.”; see at least [0038] of Mullins_371 “In another example embodiment, the sensors 102 may also include: an optical sensor (e.g., a charged-coupled device (CCD)), an orientation sensor (e.g., gyroscope), and/or an audio sensor (e.g., a microphone). For example, the device 100 may include a front-facing camera for tracking eyes movement and facial expression of the user, and a rear-facing camera for capturing a picture or a video of a physical object (or another displayed virtual object). It is noted that the sensors 102 described herein are for illustration purposes and the sensors 102 are thus not limited to the one described. In another example, sensors 102 may not be physically connected to the device 100 but are instead coupled to the device 100 via wireless means such as Wi-Fi and Bluetooth®.”) In addition, the same motivation is used as the rejection for claim 1.
Regarding claim 5, Abovitz, Mullins_371 and KAMHI teach the method of claim 1, wherein the virtual content comprises a virtual menu (see at least [0626] of Abovitz” As illustrated in FIG. 61E, the AR system renders a comprehensive virtual dashboard menu user interface, for example rendering images to the retina of the user's eyes. The virtual dashboard menu user interface may have a generally annular layout or configuration, at least partially surrounding the user, with various user selectable virtual icons spaced to be within arm's reach of the user.”; [0654] As illustrated in FIG. 65C, in response the user selection, the AR system may also render a virtual tablet type user interface tool, which provides a more detailed virtual navigation menu than the high level virtual navigation menu. The more detailed virtual navigation menu may include some or all of the menu options of the high level virtual navigation menu, as well as additional options (e.g., retrieve additional content, play interactive game associated with media title or franchise, scene selection, character exploration, actor exploration, commentary). For instance, the AR system may render the detailed virtual navigation menu to, for example, appear to the user as if sitting on a top surface of a table, within arm's reach of the user.” [0092] of Mullins_371 “In another example, the device 600 may display a virtual menu of TV channels overlaid on the TV based on the state of mind of the user 601. For example, if the user is excited, the menu of TV channels may include sports channels and action movies. In another example, the user 601 may look through a transparent display of the device 600 to a radio device. Similarly, a virtual menu of music channels may be displayed over the radio device based on the state of mind of the user 601. For example, the device 600 may display a virtual menu of classical or relaxing music channels when sensors 610 indicate that the user 601 is relaxed or sleepy.”) In addition, the same motivation is used as the rejection for claim 1.
Regarding claim 6, Abovitz, Mullins_371 and KAMHI teach the method of claim 1, wherein the virtual content is further determined based on at least one of the following: an affordance of a physical object associated with the virtual content; a function of an environment of the user; a characteristic of the user; individuals present in the environment of the user; information encoded in a fiducial marker associated with the physical object; or a current or past interaction of the user with the wearable device (see at least: [0404] of Abovitz “The totem can also incorporate one or more cameras/sensors, so that no external equipment is need to track the totem. Instead, the totem will track itself and will provide its own location, orientation, and/or identification to other devices. The on-board camera are used to visually check for feature points, to perform visual tracking to detect a position, orientation, and/or movement (e.g., position, direction, distance, speed, acceleration) of the totem itself and with respect to a reference frame. In addition, sensors mounted on the totem (such as a GPS sensor or accelerometers) can be used to detect the position and location of the totem.”; [0444] FIG. 43B shows a block shaped totem 4028, according to another illustrated embodiment. The totem may have the shape of a cube with six faces, or some other three-dimensional geometric structure. The totem may have a hard outer surface or a soft outer surface. The outer surface of the totem may have texture to facilitate a sure grip by the user. The totem may have no physical keys, physical switches or physical electronics. [0445] The AR system renders a virtual user interface image in a user's field of view, so as to appear to be on the face(s) of the outer surface of the totem. Each of the faces, and corresponding virtual input prompt, may correspond to a function, category of functions, and/or category of content or media types, tools and/or applications [0567] As mentioned briefly above, many user scenarios may involve the AR system identifying real-world activities and automatically performing actions and/or displaying virtual content based on the detected real-world activity. For example, as shown in FIG. 481, the AR system recognizes the user activity (e.g., cooking) and then creates a user interface that floats around the user's frame of reference providing useful information/virtual content associated with the activity. Similarly, many other uses can be envisioned, some of which will be described in user scenarios below.”; [0603] As illustrated in FIG. 60A, the user executes a first gesture (illustrated by double headed arrow), to open an icon based cluster user interface virtual construct (FIG. 60B). The gesture may include movement of the user's arms and/or hands or other parts of the user's body, for instance head pose or eyes. Alternatively, the user may use spoken commands to access the icon based cluster user interface virtual construct (FIG. 60B). If a more comprehensive menu is desired, the user may use a different gesture.”; [0661] In particular, FIG. 66A shows mother with her daughter in tow, pushing a shopping cart from an entrance of a grocery store. The AR system recognizes the presence of a shopping cart or a hand on the shopping cart, and determines a location of the user and/or shopping cart. Based on such, the AR system automatically launches a set of relevant applications, rendering respective user interfaces of the applications to the user's field of view. In other words, similar to the process flow of FIG. 55, the AR system recognizes the specific activity as shopping, and automatically retrieves data associated with the relevant applications to be displayed in a floating user interface.”; [0653] As illustrated in FIG. 65C, in response a user selection, the AR system renders a display of the selected entertainment or media content, and/or associated virtual menus (e.g., high level virtual navigation menu, for instance a navigation menu that allows selection of primary feature, episode, of extras materials). For example, the AR system may render a display of the selected entertainment or media content to the retina of the user's eyes, so that the selected entertainment or media content appears in the field of view of the user as if displayed on a wall of the physical space. As illustrated in FIG. 65C, the display of the selected entertainment or media content may replace at least a portion of the first virtual decor. [0654] As illustrated in FIG. 65C, in response the user selection, the AR system may also render a virtual tablet type user interface tool, which provides a more detailed virtual navigation menu than the high level virtual navigation menu. The more detailed virtual navigation menu may include some or all of the menu options of the high level virtual navigation menu, as well as additional options (e.g., retrieve additional content, play interactive game associated with media title or franchise, scene selection, character exploration, actor exploration, commentary). For instance, the AR system may render the detailed virtual navigation menu to, for example, appear to the user as if sitting on a top surface of a table, within arm's reach of the user.”; see at least [0085] of Mullins_371 “The building 608 may include points of interest icons 618, 620, 622. For example, the point of interest icon 618 may provide additional information corresponding to the location of the point of interest icon 618 relative the three-dimensional model of the building 608, when the point of interest icon 618 is triggered or otherwise selected. [0086] In one embodiment, a state of the point of interest icon 618 may be changed to in response to the user 601 viewing the point of interest 618 and a state of mind of the user 601. For example, the point of interest icon 618 may change color based on the state of mind of the user 601. [0087] In another embodiment, the device 600 may display more information about the point of interest icon 618 in response to determining that the user 601 is looking at the point of interest icon 618 and that the state of mind of the user 601 corresponds to a focused state. For example, a description box may pop up in the display 608 when the user 601 is looking at the point of interest icon 618 and the user 601 is determined to be in a focused state using sensors 610.”; [0022] of KAMHI “FIG. 1 graphically represents an example of augmented reality according to one embodiment. In this example, a user 110 starts an AR module (not shown) on a handheld device 112 (e.g., computer tablet) and is presented with a suggested list of virtual 3D models. The list may be based on the user's location, mood, and/or user profile. For example, if the AR module determines that the user 110 is located in corn field that appears as a “sea” of corn stalks 114, and that the user 110 is in a playful mood, the list may include a virtual 3D model comprising a “flying whale” configured to playfully dive in and out of the corn stalks 114. After selecting the virtual 3D model from the list, the user 110 captures a micro video clip by positioning a world facing 3D camera of the handheld device 112 (e.g., tablet) to a view of interest (e.g., an area in and above the corn field). The AR module displays a virtual image 116 of the flying whale on a screen of the handheld device 112. The virtual image 116 of the flying whale blends into the user's environment and flies in the 3D camera's field of view as if part of the user's real environment. Those skilled in the art will recognize from the disclosure herein that unlike the user 110 shown in FIG. 1, the whale image 118 does not actually exist in the surrounding environment. Rather, the whale image 118 is shown in FIG. 1 to represent the augmented reality experienced by the user 110 when viewing a video of the virtual image 116 of the flying whale through the screen on the handheld device 112.). In addition, the same motivation is used as the rejection for claim 1.
Regarding claim 7, Abovitz, Mullins_371 and KAMHI teach the method of claim 6, wherein the affordance of the physical object comprises a relation between the physical object and the environment of the user that affords an opportunity for an action or use associated with the physical object (see at least: [0404] of Abovitz “The totem can also incorporate one or more cameras/sensors, so that no external equipment is need to track the totem. Instead, the totem will track itself and will provide its own location, orientation, and/or identification to other devices. The on-board camera are used to visually check for feature points, to perform visual tracking to detect a position, orientation, and/or movement (e.g., position, direction, distance, speed, acceleration) of the totem itself and with respect to a reference frame. In addition, sensors mounted on the totem (such as a GPS sensor or accelerometers) can be used to detect the position and location of the totem.”; [0444] FIG. 43B shows a block shaped totem 4028, according to another illustrated embodiment. The totem may have the shape of a cube with six faces, or some other three-dimensional geometric structure. The totem may have a hard outer surface or a soft outer surface. The outer surface of the totem may have texture to facilitate a sure grip by the user. The totem may have no physical keys, physical switches or physical electronics. [0445] The AR system renders a virtual user interface image in a user's field of view, so as to appear to be on the face(s) of the outer surface of the totem. Each of the faces, and corresponding virtual input prompt, may correspond to a function, category of functions, and/or category of content or media types, tools and/or applications. [0446] The AR system detects or captures a user's interaction with the totem. For example, the AR system may employ one or more front facing cameras to detect a position, orientation, and/or movement (e.g., rotational direction, magnitude of rotation, angular speed, angular acceleration) of the totem with respect to some reference frame (e.g., reference frame of the real world, physical room, user's body, user's head) (similar to exemplary process flow diagram of FIG. 39). For instance, the AR system may detect one or more static orientations or a change in orientation of the totem. The AR system may also employ the front facing camera(s) to detect interactions (e.g., tap, double tap, short tap, long tap, fingertip grip, enveloping grasp) of a user's fingers with outer surface of the totem. The AR system maps the orientation and/or change in orientation (e.g., distance, direction, speed, acceleration) of the totem to user selections or inputs. The AR system optionally maps user interactions (e.g., number of interactions, types of interactions, duration of interactions) with the outer surface of the totem, and hence with various inputs (e.g., controls, functions). In response to the orientations, changes in position (e.g., movements) and/or interactions, the AR system may cause corresponding inputs to be provided to a computer or some other device.”; [0647] As illustrated in FIG. 65A, the AR system may render a user interface tool which includes a number of pre-mapped menus. For instance, the AR system may render a number of poster-like virtual images corresponding to respective pieces of entertainment or media content (e.g., movies, sports events), from which the user can select via one or more pointing gestures. The AR system may render the poster-like virtual images to, for example, appear to the user as if hanging or glued to a physical wall of the living room. Again, the AR system detects the map coordinates of the room, and displays the virtual posters in the right size and at the right orientation with respect to the mapped coordinates, such that the posters appear to be placed on the wall of the room. [0653] As illustrated in FIG. 65C, in response a user selection, the AR system renders a display of the selected entertainment or media content, and/or associated virtual menus (e.g., high level virtual navigation menu, for instance a navigation menu that allows selection of primary feature, episode, of extras materials). For example, the AR system may render a display of the selected entertainment or media content to the retina of the user's eyes, so that the selected entertainment or media content appears in the field of view of the user as if displayed on a wall of the physical space. As illustrated in FIG. 65C, the display of the selected entertainment or media content may replace at least a portion of the first virtual decor. [0654] As illustrated in FIG. 65C, in response the user selection, the AR system may also render a virtual tablet type user interface tool, which provides a more detailed virtual navigation menu than the high level virtual navigation menu. The more detailed virtual navigation menu may include some or all of the menu options of the high level virtual navigation menu, as well as additional options (e.g., retrieve additional content, play interactive game associated with media title or franchise, scene selection, character exploration, actor exploration, commentary). For instance, the AR system may render the detailed virtual navigation menu to, for example, appear to the user as if sitting on a top surface of a table, within arm's reach of the user.”; [0085] of Mullins_371 “The building 608 may include points of interest icons 618, 620, 622. For example, the point of interest icon 618 may provide additional information corresponding to the location of the point of interest icon 618 relative the three-dimensional model of the building 608, when the point of interest icon 618 is triggered or otherwise selected. [0086] In one embodiment, a state of the point of interest icon 618 may be changed to in response to the user 601 viewing the point of interest 618 and a state of mind of the user 601. For example, the point of interest icon 618 may change color based on the state of mind of the user 601. [0087] In another embodiment, the device 600 may display more information about the point of interest icon 618 in response to determining that the user 601 is looking at the point of interest icon 618 and that the state of mind of the user 601 corresponds to a focused state. For example, a description box may pop up in the display 608 when the user 601 is looking at the point of interest icon 618 and the user 601 is determined to be in a focused state using sensors 610.”; [0022] of KAMHI “FIG. 1 graphically represents an example of augmented reality according to one embodiment. In this example, a user 110 starts an AR module (not shown) on a handheld device 112 (e.g., computer tablet) and is presented with a suggested list of virtual 3D models. The list may be based on the user's location, mood, and/or user profile. For example, if the AR module determines that the user 110 is located in corn field that appears as a “sea” of corn stalks 114, and that the user 110 is in a playful mood, the list may include a virtual 3D model comprising a “flying whale” configured to playfully dive in and out of the corn stalks 114. After selecting the virtual 3D model from the list, the user 110 captures a micro video clip by positioning a world facing 3D camera of the handheld device 112 (e.g., tablet) to a view of interest (e.g., an area in and above the corn field). The AR module displays a virtual image 116 of the flying whale on a screen of the handheld device 112. The virtual image 116 of the flying whale blends into the user's environment and flies in the 3D camera's field of view as if part of the user's real environment. Those skilled in the art will recognize from the disclosure herein that unlike the user 110 shown in FIG. 1, the whale image 118 does not actually exist in the surrounding environment. Rather, the whale image 118 is shown in FIG. 1 to represent the augmented reality experienced by the user 110 when viewing a video of the virtual image 116 of the flying whale through the screen on the handheld device 112.). In addition, the same motivation is used as the rejection for claim 1.
Regarding claim 8, Abovitz, Mullins_371 and KAMHI teach the method of claim 6, wherein the affordance of the physical object is based at least partly on one or more of the following: a function of the physical object, an orientation, a type, a location, a shape, a size, or the environment of the user in which the physical object is located (see at least: [0404] of Abovitz “The totem can also incorporate one or more cameras/sensors, so that no external equipment is need to track the totem. Instead, the totem will track itself and will provide its own location, orientation, and/or identification to other devices. The on-board camera are used to visually check for feature points, to perform visual tracking to detect a position, orientation, and/or movement (e.g., position, direction, distance, speed, acceleration) of the totem itself and with respect to a reference frame. In addition, sensors mounted on the totem (such as a GPS sensor or accelerometers) can be used to detect the position and location of the totem.”; [0444] FIG. 43B shows a block shaped totem 4028, according to another illustrated embodiment. The totem may have the shape of a cube with six faces, or some other three-dimensional geometric structure. The totem may have a hard outer surface or a soft outer surface. The outer surface of the totem may have texture to facilitate a sure grip by the user. The totem may have no physical keys, physical switches or physical electronics. [0445] The AR system renders a virtual user interface image in a user's field of view, so as to appear to be on the face(s) of the outer surface of the totem. Each of the faces, and corresponding virtual input prompt, may correspond to a function, category of functions, and/or category of content or media types, tools and/or applications [0647] As illustrated in FIG. 65A, the AR system may render a user interface tool which includes a number of pre-mapped menus. For instance, the AR system may render a number of poster-like virtual images corresponding to respective pieces of entertainment or media content (e.g., movies, sports events), from which the user can select via one or more pointing gestures. The AR system may render the poster-like virtual images to, for example, appear to the user as if hanging or glued to a physical wall of the living room. Again, the AR system detects the map coordinates of the room, and displays the virtual posters in the right size and at the right orientation with respect to the mapped coordinates, such that the posters appear to be placed on the wall of the room. [0653] As illustrated in FIG. 65C, in response a user selection, the AR system renders a display of the selected entertainment or media content, and/or associated virtual menus (e.g., high level virtual navigation menu, for instance a navigation menu that allows selection of primary feature, episode, of extras materials). For example, the AR system may render a display of the selected entertainment or media content to the retina of the user's eyes, so that the selected entertainment or media content appears in the field of view of the user as if displayed on a wall of the physical space. As illustrated in FIG. 65C, the display of the selected entertainment or media content may replace at least a portion of the first virtual decor. [0654] As illustrated in FIG. 65C, in response the user selection, the AR system may also render a virtual tablet type user interface tool, which provides a more detailed virtual navigation menu than the high level virtual navigation menu. The more detailed virtual navigation menu may include some or all of the menu options of the high level virtual navigation menu, as well as additional options (e.g., retrieve additional content, play interactive game associated with media title or franchise, scene selection, character exploration, actor exploration, commentary). For instance, the AR system may render the detailed virtual navigation menu to, for example, appear to the user as if sitting on a top surface of a table, within arm's reach of the user.”; [0091]of Mullins_371 “Once the device 600 identifies that the recognized physical object or the part of the recognized physical object corresponds to a pre-identified physical object or pre-identified part of the physical object, the device may trigger a corresponding action (e.g., sending an email, generating a sound, etc.) based on the state of mind of the user 601. For example, the device 600 detects the user 601 looking through the transparent display to a bottom portion of a television set. The device 600 recognizes the television set and determines that the bottom portion of the television set (being looked at by the user 601) is associated with an action corresponding to generating a communication to the television set to switch the TV on or off. If the user 601 has looked at the bottom portion of the television set for at least several seconds and the state of mind indicates that the user is focused, the device 600 generates a corresponding signal to turn on or off the television set.”; [0022] of KAMHI “FIG. 1 graphically represents an example of augmented reality according to one embodiment. In this example, a user 110 starts an AR module (not shown) on a handheld device 112 (e.g., computer tablet) and is presented with a suggested list of virtual 3D models. The list may be based on the user's location, mood, and/or user profile. For example, if the AR module determines that the user 110 is located in corn field that appears as a “sea” of corn stalks 114, and that the user 110 is in a playful mood, the list may include a virtual 3D model comprising a “flying whale” configured to playfully dive in and out of the corn stalks 114. After selecting the virtual 3D model from the list, the user 110 captures a micro video clip by positioning a world facing 3D camera of the handheld device 112 (e.g., tablet) to a view of interest (e.g., an area in and above the corn field). The AR module displays a virtual image 116 of the flying whale on a screen of the handheld device 112. The virtual image 116 of the flying whale blends into the user's environment and flies in the 3D camera's field of view as if part of the user's real environment. Those skilled in the art will recognize from the disclosure herein that unlike the user 110 shown in FIG. 1, the whale image 118 does not actually exist in the surrounding environment. Rather, the whale image 118 is shown in FIG. 1 to represent the augmented reality experienced by the user 110 when viewing a video of the virtual image 116 of the flying whale through the screen on the handheld device 112.). In addition, the same motivation is used as the rejection for claim 1.
Regarding claim 12, Abovitz, Mullins_371 and KAMHI teach the method of claim 6, wherein the characteristic of the user comprises one or more of the following: an age, a gender, an educational level, an occupation, or a preference (see at least: [0567] of Abovitz “As mentioned briefly above, many user scenarios may involve the AR system identifying real-world activities and automatically performing actions and/or displaying virtual content based on the detected real-world activity. For example, as shown in FIG. 481, the AR system recognizes the user activity (e.g., cooking) and then creates a user interface that floats around the user's frame of reference providing useful information/virtual content associated with the activity. Similarly, many other uses can be envisioned, some of which will be described in user scenarios below.”; [0578] As illustrated the AR system renders a virtual room or virtual space in the form of a virtual office, in which the user performs their occupation or job. Hence, the virtual office is populated with various virtual tools or applications useful in performing the user's job. This may be based on received inputs by the user, based on which the AR system may retrieve data from the cloud and display the virtual tools to the user.”[0661] In particular, FIG. 66A shows mother with her daughter in tow, pushing a shopping cart from an entrance of a grocery store. The AR system recognizes the presence of a shopping cart or a hand on the shopping cart, and determines a location of the user and/or shopping cart. Based on such, the AR system automatically launches a set of relevant applications, rendering respective user interfaces of the applications to the user's field of view. In other words, similar to the process flow of FIG. 55, the AR system recognizes the specific activity as shopping, and automatically retrieves data associated with the relevant applications to be displayed in a floating user interface”; [0681] In particular, FIG. 68A shows a surgeon and surgical team, including a virtually rendered consulting or visiting surgeon, conducting a pre-operative planning session for an upcoming mitral valve replacement procedure. Each of the health care providers is wearing a respective individual AR system.”; [0028] of Mullins_371 In another example embodiment, a storage device includes a database that stores visual references, virtual objects associated with a combination of a visual reference and corresponding intensity and patterns of electric waves from the user, and features of the virtual objects. Features of the virtual objects are configured to change state in response to the intensity of the output. [0052] “The pattern module 206 measures a pattern of a combination of EEG electrodes or sensors. For example, a user may generate a unique brainwave pattern when the user is thinking about a specific car. The pattern module 206 may record the pattern of the outputs of the sensors 102 and associate the unique pattern from the user with the specific car. In another embodiment, the pattern module 206 identifies the pattern based on one or more outputs from selected or most relevant sensors. For example, the pattern module 206 may identify a unique combination of brain wave pattern when the user is looking at a car in the display. The pattern module 206 determines that the user is looking at the car using an optical sensor or a camera of the device 100. Outputs from electric waves associated with muscles of a user may also be used in combination with brain EEG to further increase the identification of unique pattern (e.g., alpha brain wave pattern A, delta bran wave pattern B, facial muscle pattern C, and eyes looking at object D in the display). As such, each unique pattern from the user may be associated with a unique visualization of a virtual object experience. In other words, a unique pattern of sensors 102 triggers an identification of a specific virtual object to be visualized in the display 104. For example, pattern xyz with intensity x′y′z′ from sensors 102 corresponds to an airplane. Pattern abc with intensity a′b′c′ from sensors 102 corresponds to moving the airplane upwards.”; [0014] of KAMHI “The RGB-D camera and AR module allow the mobile device to create instant contextual video clips that can be shared by the user. Contextual augmentation embodiments based on a user's location may allow the mobile device to access virtual 3D models from a local database or an online store that is adaptive to the user's location. For example, virtual 3D models of historic people or objects may be automatically selected or suggested to the user based on the user's location (e.g., virtual 3D models of historic people related to the construction of Louvre may be automatically selected or suggested to the user when a micro video clip is captured in the vicinity of Louvre). Other context data may also be used to select or suggest virtual 3D models, such as such as environment information (e.g., urban vs. rural setting) and/or user profiles (e.g., indicating a birthday, preferences, and/or other user data[0043] Example 3 includes the subject matter of Example 2, wherein the context data further comprises user profile information including one or more data field comprising birthdate, anniversary, animal preference, historic person preference, current celebrity preference, and genre preference.”) In addition, the same motivation is used as the rejection for claim 1.
Regarding claim 13, Abovitz, Mullins_371 and KAMHI teach the method of claim 12, wherein the preference is based at least partly on a previous usage pattern of the user, wherein the previous usage pattern comprises information on a location or a time for which the virtual content is used ( see at least: [0579] of Abovitz “As FIGS. 56A and 57 illustrate, a virtual office may be portable, being renderable in various different physical environments. It thus may be particularly advantageous if the virtual office renders identically in a subsequent use to its appearance or layout as the virtual office appeared in a most previous use or rendering. Thus, in each subsequent use or rendering, the same virtual objects will appear and the various virtual objects may retain their same spatial positions relative to one another as in a most recently previous rendering of the virtual office. [0580] In some implementations, this consistency or persistence of appearance or layout from one use to next subsequent use, may be independent of the physical environments in which the virtual space is rendered. Thus, moving from a first physical environment (e.g., physical office space) to a second physical environment (e.g., physical living room) will not affect an appearance or layout of the virtual office”; [0028] of Mullins_371 In another example embodiment, a storage device includes a database that stores visual references, virtual objects associated with a combination of a visual reference and corresponding intensity and patterns of electric waves from the user, and features of the virtual objects. Features of the virtual objects are configured to change state in response to the intensity of the output. [0052] “The pattern module 206 measures a pattern of a combination of EEG electrodes or sensors. For example, a user may generate a unique brainwave pattern when the user is thinking about a specific car. The pattern module 206 may record the pattern of the outputs of the sensors 102 and associate the unique pattern from the user with the specific car. In another embodiment, the pattern module 206 identifies the pattern based on one or more outputs from selected or most relevant sensors. For example, the pattern module 206 may identify a unique combination of brain wave pattern when the user is looking at a car in the display. The pattern module 206 determines that the user is looking at the car using an optical sensor or a camera of the device 100. Outputs from electric waves associated with muscles of a user may also be used in combination with brain EEG to further increase the identification of unique pattern (e.g., alpha brain wave pattern A, delta bran wave pattern B, facial muscle pattern C, and eyes looking at object D in the display). As such, each unique pattern from the user may be associated with a unique visualization of a virtual object experience. In other words, a unique pattern of sensors 102 triggers an identification of a specific virtual object to be visualized in the display 104. For example, pattern xyz with intensity x′y′z′ from sensors 102 corresponds to an airplane. Pattern abc with intensity a′b′c′ from sensors 102 corresponds to moving the airplane upwards”; [0020] of KAMHI “In one embodiment, a method for generating contextually augmented video includes starting a video clip capture and selecting from a group of virtual 3D models that are proposed based on location, user mood, and/or user profile (e.g., birthday, anniversary, etc.). The virtual 3D models include animations that may be automatically adapted to location, size, light, etc. The method further includes detecting people and objects in the scene and tracking skeleton and coarse gestures based on depth information, blob detection, and basic scene depth detection. The method uses the depth information to render the virtual 3D model in a photorealistic manner that supports occlusion. Thus, there is no need for a marker to be pre-located in the real scene. The method augments the real people and objects in sync with their real-time dynamic positions based on their movements (e.g., virtual butterflies follow the gestures of the people in the field of view of the world facing camera). In addition, the same motivation is used as the rejection for claim 1.
Regarding claim 14, Abovitz, Mullins_371 and KAMHI teach the method of claim 6, wherein the current interaction comprises a telepresence session between the user of the wearable device and another user (see at least [0267] of Abovitz” The passable world model essentially allows a user to effectively pass over a piece of the user's world (i.e., ambient surroundings, interactions, etc.) to another user. Each user's respective individual AR system (e.g., Sensorywear.TM. augmented reality devices) captures information as the user passes through or inhabits an environment, which the AR system processes to produce a passable world model. The individual AR system may communicate or pass the passable world model to a common or shared collection of data, referred to as the cloud. The individual AR system may communicate or pass the passable world model to other users, either directly or via the cloud. The passable world model provides the ability to efficiently communicate or pass information that essentially encompasses at least a field of view of a user. In one embodiment, the system uses the pose and orientation information, as well as collected 3D points described above in order to create the passable world.”; [0681] In particular, FIG. 68A shows a surgeon and surgical team, including a virtually rendered consulting or visiting surgeon, conducting a pre-operative planning session for an upcoming mitral valve replacement procedure. Each of the health care providers is wearing a respective individual AR system. [0685] As illustrated in FIG. 68B, the surgeon is able to reference the pre-mapped 3D anatomy (e.g., heart) during the procedure. Being able to reference the anatomy in real time, may for example, improve placement accuracy of a valve repair. Outward pointed cameras capture image information from the procedure, allowing a medical student to observe virtually via the AR system from her remote classroom. The AR system makes a patient's information readily available, for example to confirm the pathology, and avoid any critical errors.” [0028] of KAMHI “The examples shown in FIGS. 1, 2, 3A, 3B, 4A, and 4B include the use of a world facing 3D camera where the objective lens of the camera is located on one side of a mobile device and the display screen is located on another (e.g., opposite) side of the mobile device. User facing (selfie mode) 3D cameras can also be used, according to certain embodiments where the objective lens of the camera and the display screen are located on the same side of the mobile device. With a user facing 3D camera, users can experience augmented interaction with themselves in real-time at the time of the video capture.”) In addition, the same motivation is used as the rejection for claim 1.
Regarding claim 15, Abovitz, Mullins_371 and KAMHI teach the method of claim 1, wherein the wearable device comprises an augmented reality system (see at least: [0656] of Abovits “Referring now to FIGS. 66A-66J (scenes 6102-6120), another user scenario is illustrated. FIGS. 66A-66J illustrate an AR system implemented retail experience, according to one illustrated embodiment. [0657] As illustrated, a mother and daughter each wearing respective individual AR systems receive an augmented reality experience while shopping in a retail environment, for example a supermarket. As explained herein, the AR system may provide entertainment as well as facilitate the shopping experience. For example, the AR system may render virtual content, for instance virtual characters which may appear to jump from a box or carton, and/or offer virtual coupons for selected items. The AR system may render games, for example games based on locations throughout the store and/or based on items on shopping list, list of favorites, or a list of promotional items. The augmented reality environment encourages children to play, while moving through each location at which a parent or accompanying adult needs to pick up an item. Even adults may play.”; Fig.1 of Mullins_371; [0029] of KAMHI “FIG. 5 is a block diagram of an augmented reality device 500 according to one embodiment. The augmented reality device 500 may be embodied as any type of device configured to render one or more augmented reality objects or otherwise perform the functions described herein, including, without limitation, a smartphone, a cellular telephone, a handset, a tablet computer, a laptop computer, a notebook computer, a messaging device, a gaming device, a smart appliance, a network appliance, a web appliance, a multiprocessor system, and/or a mobile consumer electronic device.”) In addition, the same motivation is used as the rejection for claim 1.
Regarding independent claim 16, Abovitz teaches a wearable device for generating virtual content in a three-dimensional (3D) environment of a user (see at least Figs 61A-61E), the wearable device comprising:
an augmented reality display configured to present the virtual content in a 3D environment to a user (see at least at [0612] FIG. 61B (scene 6104) shows a user sitting in a physical living room space, and using an AR system to experience a virtual room or virtual space in the form of a virtual entertainment or media room, the user executing gestures to interact with a user interface virtual construct, according to one illustrated embodiment.[0613] The physical living room may include one or more physical objects, for instance walls, floor, ceiling, a coffee table and sofa. As previously noted, the user may wear a head worn AR system, or head worn component of an AR system, operable to render virtual content in a field of view of the user. For example, the head worn AR system or component may render virtual objects, virtual tools and applications onto the retina of each eye of the user.”);
a sensor configured to measure a physiological parameter of the user (see at least at [0335] Referring to process flow diagram 3000 of FIG. 30, on a basic level, the AR system is configured to receive input (e.g., visual input 2202 from the user's wearable system, input from room cameras 2204, sensory input 2206 in the form of various sensors in the system, gestures, totems, eye tracking etc.) from one or more AR systems. The AR systems may constitute one or more user wearable systems, and/or stationary room systems (room cameras, etc). The wearable AR systems not only provide images from the FOV cameras, they may also be equipped with various sensors (e.g., accelerometers, temperature sensors, movement sensors, depth sensors, GPS, etc.), as discussed above, to determine the location, and various other attributes of the environment of the user. Of course, this information may further be supplemented with information from stationary cameras mentioned previously that may provide images and/or various cues from a different point of view. It should be appreciated that image data may be reduced to a set of points, as explained above.”; ([0672] In some implementations, the AR system may capture images of the customers, for example via inward facing cameras carried by each customer's individual head worn component. The AR system may provide a composited virtual image to the celebrity of a crowd composed of the various customers.”);
a hardware processor in communication with the sensor and the augmented reality display( see at least at [0236] Referring to FIG. 15, one embodiment of the head-worn AR system has a suitable user display device (14) as shown in FIG. 15. The user display device may comprise a display lens (82) which may be mounted to a user's head or eyes by a housing or frame (84).”; [0237]”… The depicted system (14) also comprises a head pose processor (36), such as an ASIC (application specific integrated circuit), FPGA (field programmable gate array), and/or ARM processor (advanced reduced-instruction-set machine), which may be configured to calculate real or near-real time user head pose from wide field of view image information output from the capture devices (16). Also shown is another processor (32) configured to execute digital and/or analog processing to derive pose from the gyro, compass, and/or accelerometer data from the sensor assembly (39).”), the hardware processor programmed to:
determine a current location of the user (0017] In another aspect, a method of rendering virtual content to a user is disclosed. The method comprises detecting a location of a user, retrieving a set of data associated with a part of a virtual world model that corresponds to the detected location of the user, wherein the virtual world model comprises data associated with a set of map points of the real world, and rendering, based on the set of retrieved data, virtual content to a user device of the user, such that the virtual content, when viewed by the user, appears to be placed in relation to a set of physical objects in a physical environment of the user.”; [0273] FIG. 21 illustrates an exemplary method 2100 of interacting with the passable world model. First, the user's individual AR system may detect a location of the user (step 2102). The location may be derived by the topological map of the system, as will be described in further detail below. The location may be derived by GPS or any other localization tool.”; [0640] FIG. 63 shows a user sitting in a physical living room space, and using an AR system to experience a virtual room or virtual space in the form of a virtual entertainment or media room, the user executing gestures to interact with a user interface virtual construct to provide input by proxy, according to one illustrated embodiment.”);
obtain, using the sensor, data associated with the physiological parameter of the user(see at least at [0294] More particularly, the user's individual AR system contains information about the user's head pose and orientation in a space, information about hand movement etc. of the user, information about the user's eyes and eye gaze, information about any totems that are being used by the user. Thus, the user's individual AR system already holds a lot of information about the user's interaction within a particular space that is transmitted to the passable world model. This information may then be reliably used to create avatars for the user and help the avatar communicate with other avatars or users of that space. It should be appreciated that no third party cameras are needed to animate the avatar, rather, the avatar is animated based on the user's individual AR system.”; ([0294] More particularly, the user's individual AR system contains information about the user's head pose and orientation in a space, information about hand movement etc. of the user, information about the user's eyes and eye gaze, information about any totems that are being used by the user. Thus, the user's individual AR system already holds a lot of information about the user's interaction within a particular space that is transmitted to the passable world model. This information may then be reliably used to create avatars for the user and help the avatar communicate with other avatars or users of that space. It should be appreciated that no third party cameras are needed to animate the avatar, rather, the avatar is animated based on the user's individual AR system.” ( [0345] With regard to the camera systems, the depicted configuration shows three pairs of cameras: a relative wide field of view ("FOV") or "passive SLAM" pair of cameras arranged to the sides of the user's face, a different pair of cameras oriented in front of the user to handle the Stereo imaging process and also to capture hand gestures and totem/object tracking in front of the user's face. Then there is a pair of Eye Cameras oriented into the eyes of the user so they may attempt to triangulate eye vectors and other information. As noted above, the system may also comprise one or more textured light projectors (such as infrared, or "IR", projectors) to inject texture into a scene.”; [0370] Because the AR system is configured to continuously "know" the physical location and orientation of the user's surroundings, and given that the AR system is constantly collecting various types of data regarding the user's environment (e.g., FOV images, eye tracking data, sensory data, audio data, etc.) conventional types of user inputs may not be necessary. For example, rather than the user physically pressing a button or explicitly speaking a command, user input in the AR system may be automatically recognized. For example, the system may automatically recognize a gesture made by the user's fingers. In another example, the AR system may recognize an input based on eye tracking data. Or, in another example, the AR system may recognize a location, and automatically use that as user input to display virtual content. One important type of user input is gesture recognition in order to perform an action or display virtual content, as will be described below.”);
identify, based at least partly on the data associated with the physiological parameter of the user, a physiological state of the user (see at least at [0596] To allow user selection of and/or navigation between virtual rooms or virtual spaces, the AR system may be responsive to one or more of, for instance, gestures, voice commands, eye tracking, and/or selection of physical buttons, keys or switches for example carried by a head worn component, belt pack or other physical structure of the individual AR system. The user input may be indicative of a direct selection of a virtual space or room, or may cause a rendering of a menu or submenus to allow user selection of a virtual space or room. [0409] Thus a totem may be any object on which virtual content can be rendered, including for example a body part (e.g., hand) to which virtual content can be locked in a user experience (UX) context. In some implementations, the AR system can render virtual content so as to appear to be coming out from behind a totem, for instance appearing to emerge from behind a user's hand, and slowly wrapping at least partially around the user's hand. The AR system detects user interaction with the virtual content, for instance user finger manipulation with the virtual content which wrapped partially around the user's hand. Alternatively, the AR system may render virtual content so as to appear to emerge from a palm of the user's hand, and detection user fingertip interaction or manipulate of that virtual content. Thus, the virtual content may be locked to a reference from of a user's hand. The AR system may be responsive to various user interactions or gestures, including looking at some item of virtual content, moving hands, touching hands to themselves or to the environment, other gestures, opening and/or closing eyes, etc.”)
evaluate potential virtual content based at least in part on the current location of the user (see at least [0409] Thus a totem may be any object on which virtual content can be rendered, including for example a body part (e.g., hand) to which virtual content can be locked in a user experience (UX) context. In some implementations, the AR system can render virtual content so as to appear to be coming out from behind a totem, for instance appearing to emerge from behind a user's hand, and slowly wrapping at least partially around the user's hand. The AR system detects user interaction with the virtual content, for instance user finger manipulation with the virtual content which wrapped partially around the user's hand. Alternatively, the AR system may render virtual content so as to appear to emerge from a palm of the user's hand, and detection user fingertip interaction or manipulate of that virtual content. Thus, the virtual content may be locked to a reference from of a user's hand. The AR system may be responsive to various user interactions or gestures, including looking at some item of virtual content, moving hands, touching hands to themselves or to the environment, other gestures, opening and/or closing eyes, etc.” [0602] The physical living room may include one or more physical objects, for instance walls, floor, ceiling, a coffee table and sofa. As previously noted, the user may wear a head worn AR system, or head worn component of an AR system, operable to render virtual content in a field of view of the user. For example, the head worn AR system or component may render virtual objects, virtual tools and applications onto the retina of each eye of the user. [0603] As illustrated in FIG. 60A, the user executes a first gesture (illustrated by double headed arrow), to open an icon based cluster user interface virtual construct (FIG. 60B). The gesture may include movement of the user's arms and/or hands or other parts of the user's body, for instance head pose or eyes. Alternatively, the user may use spoken commands to access the icon based cluster user interface virtual construct (FIG. 60B). If a more comprehensive menu is desired, the user may use a different gesture.”; [0654] As illustrated in FIG. 65C, in response the user selection, the AR system may also render a virtual tablet type user interface tool, which provides a more detailed virtual navigation menu than the high level virtual navigation menu. The more detailed virtual navigation menu may include some or all of the menu options of the high level virtual navigation menu, as well as additional options (e.g., retrieve additional content, play interactive game associated with media title or franchise, scene selection, character exploration, actor exploration, commentary). For instance, the AR system may render the detailed virtual navigation menu to, for example, appear to the user as if sitting on a top surface of a table, within arm's reach of the user.”);
select first virtual content to be presented to the user from the potential virtual content based at least partly on the current location of the user; (see at least [0602] The physical living room may include one or more physical objects, for instance walls, floor, ceiling, a coffee table and sofa. As previously noted, the user may wear a head worn AR system, or head worn component of an AR system, operable to render virtual content in a field of view of the user. For example, the head worn AR system or component may render virtual objects, virtual tools and applications onto the retina of each eye of the user. [0603] As illustrated in FIG. 60A, the user executes a first gesture (illustrated by double headed arrow), to open an icon based cluster user interface virtual construct (FIG. 60B). The gesture may include movement of the user's arms and/or hands or other parts of the user's body, for instance head pose or eyes. Alternatively, the user may use spoken commands to access the icon based cluster user interface virtual construct (FIG. 60B). If a more comprehensive menu is desired, the user may use a different gesture.”; [0654] As illustrated in FIG. 65C, in response the user selection, the AR system may also render a virtual tablet type user interface tool, which provides a more detailed virtual navigation menu than the high level virtual navigation menu. The more detailed virtual navigation menu may include some or all of the menu options of the high level virtual navigation menu, as well as additional options (e.g., retrieve additional content, play interactive game associated with media title or franchise, scene selection, character exploration, actor exploration, commentary). For instance, the AR system may render the detailed virtual navigation menu to, for example, appear to the user as if sitting on a top surface of a table, within arm's reach of the user.”)
determine a spatial location for displaying the first virtual content in the 3D space; generate a virtual user interface comprising at least the first virtual content; and display to the user, via the display of the wearable device, the first virtual content at the determined spatial location(see at least [0609] As illustrated in FIG. 61A (scene 6102), the AR system may render a functional group or pod user interface virtual construct, so at to appear in a user's field of view, preferably appearing to reside within a reach of the user. The pod user interface virtual construct includes a plurality of virtual room or virtual space based applications, which conveniently provides access from one virtual room or virtual space to functional tools and applications which are logically associated with another virtual room or virtual space. The pod user interface virtual construct forms a mini work station for the user. [0610] As previously discussed, the AR system may render virtual content at any apparent or perceived depth in the virtual space. Hence, the virtual content may be rendered to appear or seem to appear at any depth in the physical space onto which the virtual space is mapped. Implementation of intelligent depth placement of various elements or instances of virtual content may advantageously prevent clutter in the user's field of view. As previously noted, the AR system may render virtual content so as to appear to be mounted or glued to a physical surface in the physical space, or may render the virtual content so as to appear to be floating in the physical space. Thus, the AR system may render the pod user interface virtual construct floating within the reach of the user, while concurrently rendering a virtual room or space (e.g., virtual entertainment or media room or space) spaced farther away for the user, for instance appear to be glued to the walls and table.. [0626] As illustrated in FIG. 61E, the AR system renders a comprehensive virtual dashboard menu user interface, for example rendering images to the retina of the user's eyes. The virtual dashboard menu user interface may have a generally annular layout or configuration, at least partially surrounding the user, with various user selectable virtual icons spaced to be within arm's reach of the user.”; [0654] As illustrated in FIG. 65C, in response the user selection, the AR system may also render a virtual tablet type user interface tool, which provides a more detailed virtual navigation menu than the high level virtual navigation menu. The more detailed virtual navigation menu may include some or all of the menu options of the high level virtual navigation menu, as well as additional options (e.g., retrieve additional content, play interactive game associated with media title or franchise, scene selection, character exploration, actor exploration, commentary). For instance, the AR system may render the detailed virtual navigation menu to, for example, appear to the user as if sitting on a top surface of a table, within arm's reach of the user.”). Abovitz is understood to be silent on the remaining limitations of claim 16.
In the same field of endeavor, Mullins_371 teaches a physiological sensor ([0025] In another example embodiment, the device also includes an EEG brain wave sensor coupled to a head of the user to generate brain wave data, a heart rate sensor to generate heart rate data of the user, an eye tracking sensor to determine a gaze of the user on the display, a location sensor to determine a geographic location of the device, and a social network module configured to access social network information associated with the user. The electric wave application identifies the state of mind of the user based on the brain wave data, heart rate data, the gaze of the user, the geographic location of the device, and social network information associated with the user.”;[0037] “The sensors 102 may include electrodes that measure electrical activity from a human. For example, the sensors 102 may include electrodes to measure EEG (electroencephalography) waves of brains, EMG (electromyography) waves of muscles, and EOG (electrooculogram) waves of eyes. The sensors 102 can be used to monitor brainwaves through EEG by detecting electrical signals about a person's level of concentration or state of mind. The sensors may be implemented, for example, by using a headset attached to a head of a user. In another example, the sensors 102 can be used to monitor facial muscles to detect facial expressions of the user”; [0038] In another example embodiment, the sensors 102 may also include: an optical sensor (e.g., a charged-coupled device (CCD)), an orientation sensor (e.g., gyroscope), and/or an audio sensor (e.g., a microphone).) configured to measure a physiological parameter of the user ([0098] In operation 1006, the virtual object generation module 304 identifies a virtual object based on the physical object (or a visual reference on the physical object) and the intensity and pattern of the electric waves of the user. In another example, other parameters may be used to identify the virtual object. For example, heart rate data, voice data may be used to identify the virtual object.”);
determine a current location of the user ([0025] In another example embodiment, the device also includes an EEG brain wave sensor coupled to a head of the user to generate brain wave data, a heart rate sensor to generate heart rate data of the user, an eye tracking sensor to determine a gaze of the user on the display, a location sensor to determine a geographic location of the device, and a social network module configured to access social network information associated with the user. The electric wave application identifies the state of mind of the user based on the brain wave data, heart rate data, the gaze of the user, the geographic location of the device, and social network information associated with the user.”) :
obtain, using the physiological sensor, data associated with the physiological parameter of the user ([0025] In another example embodiment, the device also includes an EEG brain wave sensor coupled to a head of the user to generate brain wave data, a heart rate sensor to generate heart rate data of the user, an eye tracking sensor to determine a gaze of the user on the display, a location sensor to determine a geographic location of the device, and a social network module configured to access social network information associated with the user. The electric wave application identifies the state of mind of the user based on the brain wave data, heart rate data, the gaze of the user, the geographic location of the device, and social network information associated with the user.”);
identify, based at least partly on the data associated with the physiological parameter of the user ([0098] In operation 1006, the virtual object generation module 304 identifies a virtual object based on the physical object (or a visual reference on the physical object) and the intensity and pattern of the electric waves of the user. In another example, other parameters may be used to identify the virtual object. For example, heart rate data, voice data may be used to identify the virtual object.”), a physiological state of the user(see at least [0025] of Mullins “In another example embodiment, the device also includes an EEG brain wave sensor coupled to a head of the user to generate brain wave data, a heart rate sensor to generate heart rate data of the user, an eye tracking sensor to determine a gaze of the user on the display, a location sensor to determine a geographic location of the device, and a social network module configured to access social network information associated with the user. The electric wave application identifies the state of mind of the user based on the brain wave data, heart rate data, the gaze of the user, the geographic location of the device, and social network information associated with the user.[0030] Biometrics data from the user may also be used to program a visual gesture resulting in a command or user input in the HMD. For example, the visual gesture may include detecting that a user of the HMD is staring at a physical object for more than a time duration threshold in combination with a high heart rate. Therefore, when a user's eye gaze intently directed towards an object (e.g., a switch blinking red) for more than 3 seconds, and the user's heart beat exceeds a threshold heartbeat, the HMD generates a specific AR content (e.g., instructions or operation of the switch, virtual arrows telling the user to act on the switch) without the user having to use his hands to tap on a touch sensitive surface on the HMD.”);
evaluate potential virtual content based at least in part on the physiological state of the user (see at least [0045] The storage device 108 may be configured to store a database of visual references, virtual objects corresponding to the visual references, features of the virtual objects corresponding to the virtual objects, and corresponding states of mind. The features of the virtual objects can change with the state of mind of the user. For example, the color of the virtual chair can change from blue to red as the user becomes more focused. The virtual chair may be displayed in a blue color if the user is relaxed. In another example, features of the virtual object change when the features are present in the focus area of the display 104. For example, the visual reference may include a machine-readable code or a previously identified image (e.g., a picture of shoe). The previously identified image of the show may correspond to a three-dimensional virtual shoe that can be viewed from different angles by manipulating the position of the device 100 relative to the picture of the shoe. Features of the three-dimensional virtual shoe may include selectable icons on the three-dimensional virtual shoe. An icon may be selected or activated by moving (e.g., repositioning, reorienting, or both) the device 100 to display the icon within a focus area of the display 104. For example, the focus area may be a central area of the display 104, a corner of the display 104, an edge of the display 104, or any suitable combination thereof. [0058] The virtual object generation module 304 generates and displays a visualization of a three-dimensional virtual object engaged with an image of the physical object captured by the sensor 102 of the device 100 (e.g., the virtual sofa floats and rotates on top of the magazine page). The virtual object may be based on the visual reference (e.g., a furniture ad in the magazine page) and a state of mind of the user. In one embodiment, each virtual object may be uniquely associated with a visual reference and a particular state of mind. The virtual object generation module 304 renders the visualization of the virtual object based a position of the device 100 relative to the visual reference. In another embodiment, attributes of the virtual object may be based on the state of mind of the user. For example, the virtual object generation module 304 may generate a blue color sofa when the state of mind of the user indicates that the user is relaxed and is thinking of a sofa. Similarly, the virtual object generation module 304 may generate a red color sofa when the state of mind of the user indicates that the user is excited and thinking of a sofa. [0079] The device 600 generates a visualization of a three-dimensional virtual object in a display 602 of the device 600 based on outputs from sensors 610 and the visual reference 606. For example, the device 600 may determine that the user 601 is geographically located at an architectural firm. The device 600 determines from the sensors 610 that the state of mind of the user 601 is focused on a high rise building. In another embodiment, a front facing camera 614 of the device 600 may further enhance and provide additional data on the state of mind of the user 601. For example, the device 600 may obtain a live picture of the user 601 using the front facing camera 614 to determine a smile or a frown. In another example, the front facing camera 614 may be used for facial recognition to determine the identity of the user 601. The device 600 may retrieve preferences from the user 601 such as, for example, favorite colors or items. In another example, the device 600 determines, identifies, and manipulates a virtual object to be displayed in the display 602 based on a combination of the geographic location of the device 600 (e.g., office, home, restaurant, city, country), time of capture (e.g., morning, afternoon, evening, holiday, weekend) of the visual reference 606, orientation (e.g., portrait or landscape, how close) of the device 600 relative to the visual reference 606, identification of the user 601 (e.g. using facial recognition, or login information), preferences of the user 601 (e.g., favorite color, favorite type of music) social network information (e.g, number of friends, interests, proximity of friends, postings) related to the user 601, outputs from sensors 610 (e.g., EEG brain waves, EMG muscles waves, EOG eyes waves, heart rate, blood pressure), and the visual reference 606..);
select first virtual content to be presented to the user from the potential virtual content based at least partly on the evaluation of the physiological state of the user (see at least [0045] The storage device 108 may be configured to store a database of visual references, virtual objects corresponding to the visual references, features of the virtual objects corresponding to the virtual objects, and corresponding states of mind. The features of the virtual objects can change with the state of mind of the user. For example, the color of the virtual chair can change from blue to red as the user becomes more focused. The virtual chair may be displayed in a blue color if the user is relaxed. In another example, features of the virtual object change when the features are present in the focus area of the display 104. For example, the visual reference may include a machine-readable code or a previously identified image (e.g., a picture of shoe). The previously identified image of the show may correspond to a three-dimensional virtual shoe that can be viewed from different angles by manipulating the position of the device 100 relative to the picture of the shoe. Features of the three-dimensional virtual shoe may include selectable icons on the three-dimensional virtual shoe. An icon may be selected or activated by moving (e.g., repositioning, reorienting, or both) the device 100 to display the icon within a focus area of the display 104. For example, the focus area may be a central area of the display 104, a corner of the display 104, an edge of the display 104, or any suitable combination thereof. [0079] The device 600 generates a visualization of a three-dimensional virtual object in a display 602 of the device 600 based on outputs from sensors 610 and the visual reference 606. For example, the device 600 may determine that the user 601 is geographically located at an architectural firm. The device 600 determines from the sensors 610 that the state of mind of the user 601 is focused on a high rise building. In another embodiment, a front facing camera 614 of the device 600 may further enhance and provide additional data on the state of mind of the user 601. For example, the device 600 may obtain a live picture of the user 601 using the front facing camera 614 to determine a smile or a frown. In another example, the front facing camera 614 may be used for facial recognition to determine the identity of the user 601. The device 600 may retrieve preferences from the user 601 such as, for example, favorite colors or items. In another example, the device 600 determines, identifies, and manipulates a virtual object to be displayed in the display 602 based on a combination of the geographic location of the device 600 (e.g., office, home, restaurant, city, country), time of capture (e.g., morning, afternoon, evening, holiday, weekend) of the visual reference 606, orientation (e.g., portrait or landscape, how close) of the device 600 relative to the visual reference 606, identification of the user 601 (e.g. using facial recognition, or login information), preferences of the user 601 (e.g., favorite color, favorite type of music) social network information (e.g, number of friends, interests, proximity of friends, postings) related to the user 601, outputs from sensors 610 (e.g., EEG brain waves, EMG muscles waves, EOG eyes waves, heart rate, blood pressure), and the visual reference 606..0081] FIG. 6B is a block diagram illustrating an example of a visualization of an action on a virtual object in the device based on thought, according to some example embodiments. The device 600 determines a change in the state of mind of the user 601 (e.g., the user is thinking about rain on the building). The device 600 then generates a change in the visualization of the three-dimensional virtual object in the display 602 of the device 600 based on the change in the state of mind of the user 601 in response to changes in outputs from sensors 610 and the front facing camera 614. For example, rain over the building 608 may be dynamically animated in the display 602 when the device 600 detects that the user 601 has focused on rain. [0082] As such, changes of the already displayed three-dimensional virtual object in the display 602 are determined based on the changes in the state of mind of the user 601. In another example, the color of the building 608 may change to a lighter hue when the user 601 becomes more relaxed while looking at the building 608. In another example, the texture of the building 608 may change to a rougher texture when the user 601 becomes agitated.”);
determine a location for displaying the virtual content in the 3D space (see at least at [0043] The augmented reality application 112 may generate and display a visualization of the virtual object engaged with an image or picture of the physical object in the display 104. The virtual object may be generated based on the visual reference and the state of mind of the user. Each virtual object may correspond to a unique visual reference and corresponding state of mind (e.g., unique to that virtual object within the augmented reality application 112). In another embodiment, the augmented reality application 112 renders the visualization of the virtual object based a position and an orientation of the device 100 relative to the visual reference in the physical object.; [0079] The device 600 generates a visualization of a three-dimensional virtual object in a display 602 of the device 600 based on outputs from sensors 610 and the visual reference 606. For example, the device 600 may determine that the user 601 is geographically located at an architectural firm. The device 600 determines from the sensors 610 that the state of mind of the user 601 is focused on a high rise building. In another embodiment, a front facing camera 614 of the device 600 may further enhance and provide additional data on the state of mind of the user 601”).;
generate a virtual user interface comprising at least the first virtual content (see at least at [0076] The action module 504 may be configured to trigger an action similar to tapping on the icon on the display 104. For example, the action module 504 may generate a message notification, a dialog box, a menu, or any other action triggered by the presence of the feature in the focus area in the display 104 or by the state of mind of the user. In another embodiment, the action module 404 may be configured to generate a communication from the device 100 to another device, for example, via a wireless network.”; 0092] In another example, the device 600 may display a virtual menu of TV channels overlaid on the TV based on the state of mind of the user 601. For example, if the user is excited, the menu of TV channels may include sports channels and action movies. In another example, the user 601 may look through a transparent display of the device 600 to a radio device. Similarly, a virtual menu of music channels may be displayed over the radio device based on the state of mind of the user 601. For example, the device 600 may display a virtual menu of classical or relaxing music channels when sensors 610 indicate that the user 601 is relaxed or sleepy.”) and
display to the user, via the display of the wearable device, the first virtual content at the determined location (see at least at [0081] FIG. 6B is a block diagram illustrating an example of a visualization of an action on a virtual object in the device based on thought, according to some example embodiments. The device 600 determines a change in the state of mind of the user 601 (e.g., the user is thinking about rain on the building). The device 600 then generates a change in the visualization of the three-dimensional virtual object in the display 602 of the device 600 based on the change in the state of mind of the user 601 in response to changes in outputs from sensors 610 and the front facing camera 614. For example, rain over the building 608 may be dynamically animated in the display 602 when the device 600 detects that the user 601 has focused on rain.[0082] As such, changes of the already displayed three-dimensional virtual object in the display 602 are determined based on the changes in the state of mind of the user 601. In another example, the color of the building 608 may change to a lighter hue when the user 601 becomes more relaxed while looking at the building 608. In another example, the texture of the building 608 may change to a rougher texture when the user 601 becomes agitated.) In addition, the same motivation is used as the rejection for claim 1. Both Abvitz and Mullins_371 are understood to be silent on the remaining limitations of claim 16.
In the same field of endeavor, KAMHI teaches evaluate potential virtual content based at least in part on the physiological state of the user and the current location of the user ([0022] FIG. 1 graphically represents an example of augmented reality according to one embodiment. In this example, a user 110 starts an AR module (not shown) on a handheld device 112 (e.g., computer tablet) and is presented with a suggested list of virtual 3D models. The list may be based on the user's location, mood, and/or user profile. For example, if the AR module determines that the user 110 is located in corn field that appears as a “sea” of corn stalks 114, and that the user 110 is in a playful mood, the list may include a virtual 3D model comprising a “flying whale” configured to playfully dive in and out of the corn stalks 114. After selecting the virtual 3D model from the list, the user 110 captures a micro video clip by positioning a world facing 3D camera of the handheld device 112 (e.g., tablet) to a view of interest (e.g., an area in and above the corn field). The AR module displays a virtual image 116 of the flying whale on a screen of the handheld device 112. The virtual image 116 of the flying whale blends into the user's environment and flies in the 3D camera's field of view as if part of the user's real environment. Those skilled in the art will recognize from the disclosure herein that unlike the user 110 shown in FIG. 1, the whale image 118 does not actually exist in the surrounding environment. Rather, the whale image 118 is shown in FIG. 1 to represent the augmented reality experienced by the user 110 when viewing a video of the virtual image 116 of the flying whale through the screen on the handheld device 112.” [0039] In certain embodiments, the method 800 does not require any training stage or binding of a virtual 3D model to the detection and tracking of a 2D or 3D object. The type of potential augmentation may be suggested to the user based on the user's location, mood, and/or profile. Once the type of augmentation is chosen, the rendering of the augmentation into the reality is driven by the people and the objects in the scene (e.g., within the field of view of the 3D video capture device). The augmented virtual 3D objects interact with the physical objects in the scene in a natural way. For example, if the person waves his hands, a virtual birds follows his hands, a virtual monkey climbs on to the shoulder of a person and interacts with the person as it would in real life.);
select first virtual content to be presented to the user from the potential virtual content based at least partly on the evaluation of the physiological state of the user and the current location of the user ([0022] FIG. 1 graphically represents an example of augmented reality according to one embodiment. In this example, a user 110 starts an AR module (not shown) on a handheld device 112 (e.g., computer tablet) and is presented with a suggested list of virtual 3D models. The list may be based on the user's location, mood, and/or user profile. For example, if the AR module determines that the user 110 is located in corn field that appears as a “sea” of corn stalks 114, and that the user 110 is in a playful mood, the list may include a virtual 3D model comprising a “flying whale” configured to playfully dive in and out of the corn stalks 114. After selecting the virtual 3D model from the list, the user 110 captures a micro video clip by positioning a world facing 3D camera of the handheld device 112 (e.g., tablet) to a view of interest (e.g., an area in and above the corn field). The AR module displays a virtual image 116 of the flying whale on a screen of the handheld device 112. The virtual image 116 of the flying whale blends into the user's environment and flies in the 3D camera's field of view as if part of the user's real environment. Those skilled in the art will recognize from the disclosure herein that unlike the user 110 shown in FIG. 1, the whale image 118 does not actually exist in the surrounding environment. Rather, the whale image 118 is shown in FIG. 1 to represent the augmented reality experienced by the user 110 when viewing a video of the virtual image 116 of the flying whale through the screen on the handheld device 112.”) In addition, the same motivation is used as the rejection for claim 1.
Thus, the combination of Abovitz, Mullins_371 and KAMHI teaches a wearable device for generating virtual content in a three- dimensional (3D) environment of a user, the wearable device comprising: an augmented reality display configured to present the virtual content in the 3D environment to a user; a physiological sensor configured to measure a physiological parameter of the user; a hardware processor in communication with the physiological sensor and the augmented reality display, the hardware processor programmed to: determine a current location of the user; obtain, using the physiological sensor, data associated with the physiological parameter of the user; identify, based at least partly on the data associated with the physiological parameter of the user, a physiological state of the user; evaluate potential virtual content based at least in part on the physiological state of the user and the current location of the user; select first virtual content to be presented to the user from the potential virtual content based at least partly on the evaluation of the physiological state of the user and the current location of the user; determine a spatial location for displaying the first virtual content in the 3D space; generate a virtual user interface comprising at least the first virtual content; and display to the user, via the display of the wearable device, the first virtual content at the determined spatial location.
Regarding claim 18, Abovitz, Mullins_371 and KAMHI teach the wearable device of claim 16, Remaining limitations of claim 18 is similar scope to claim 3 and therefore rejected under the same rational.
Regarding claim 19, Abovitz, Mullins_371 and KAMHI teach the wearable device of claim 16, Remaining limitations of claim 19 is similar scope to claim 6 and therefore rejected under the same rational.
Regarding claim 21, Abovitz, Mullins_371 and KAMHI teach the of method of claim 1, wherein the physiological state corresponds to the mood of the user such that the determining of the virtual content to be presented to the user is based at least partly on the mood of the user (see at least: [0409] of Abovitz Thus a totem may be any object on which virtual content can be rendered, including for example a body part (e.g., hand) to which virtual content can be locked in a user experience (UX) context. In some implementations, the AR system can render virtual content so as to appear to be coming out from behind a totem, for instance appearing to emerge from behind a user's hand, and slowly wrapping at least partially around the user's hand. The AR system detects user interaction with the virtual content, for instance user finger manipulation with the virtual content which wrapped partially around the user's hand. Alternatively, the AR system may render virtual content so as to appear to emerge from a palm of the user's hand, and detection user fingertip interaction or manipulate of that virtual content. Thus, the virtual content may be locked to a reference from of a user's hand. The AR system may be responsive to various user interactions or gestures, including looking at some item of virtual content, moving hands, touching hands to themselves or to the environment, other gestures, opening and/or closing eyes, etc.” [0598] The physical living room may include one or more physical objects, for instance walls, floor, ceiling, a coffee table and sofa. As previously noted, the user may wear a head worn AR system, or head worn component of an AR system, operable to render virtual content in a field of view of the user. For example, the head worn AR system or component may render virtual objects, virtual tools and applications onto the retina of each eye of the user.”; [0603] As illustrated in FIG. 60A, the user executes a first gesture (illustrated by double headed arrow), to open an icon based cluster user interface virtual construct (FIG. 60B). The gesture may include movement of the user's arms and/or hands or other parts of the user's body, for instance head pose or eyes. Alternatively, the user may use spoken commands to access the icon based cluster user interface virtual construct (FIG. 60B). If a more comprehensive menu is desired, the user may use a different gesture”; [0053] of Mullins_317” The state of mind identification module 210 may thus determine and identify a state of mind of the user based on the intensity module 204 and the pattern module 206. For example, the state of mind identification module 210 may determine that the user is happy, relaxed, angry, focused, hungry, or thirsty. In another embodiment, the state of mind identification module 210 may determine a specific object (e.g., a minivan) or a specific action (e.g., open a door) that the user is thinking.”; [0058] The virtual object generation module 304 generates and displays a visualization of a three-dimensional virtual object engaged with an image of the physical object captured by the sensor 102 of the device 100 (e.g., the virtual sofa floats and rotates on top of the magazine page). The virtual object may be based on the visual reference (e.g., a furniture ad in the magazine page) and a state of mind of the user. In one embodiment, each virtual object may be uniquely associated with a visual reference and a particular state of mind. The virtual object generation module 304 renders the visualization of the virtual object based a position of the device 100 relative to the visual reference. In another embodiment, attributes of the virtual object may be based on the state of mind of the user. For example, the virtual object generation module 304 may generate a blue color sofa when the state of mind of the user indicates that the user is relaxed and is thinking of a sofa. Similarly, the virtual object generation module 304 may generate a red color sofa when the state of mind of the user indicates that the user is excited and thinking of a sofa.”; [0022] of KAMHI “FIG. 1 graphically represents an example of augmented reality according to one embodiment. In this example, a user 110 starts an AR module (not shown) on a handheld device 112 (e.g., computer tablet) and is presented with a suggested list of virtual 3D models. The list may be based on the user's location, mood, and/or user profile. For example, if the AR module determines that the user 110 is located in corn field that appears as a “sea” of corn stalks 114, and that the user 110 is in a playful mood, the list may include a virtual 3D model comprising a “flying whale” configured to playfully dive in and out of the corn stalks 114. After selecting the virtual 3D model from the list, the user 110 captures a micro video clip by positioning a world facing 3D camera of the handheld device 112 (e.g., tablet) to a view of interest (e.g., an area in and above the corn field). The AR module displays a virtual image 116 of the flying whale on a screen of the handheld device 112. The virtual image 116 of the flying whale blends into the user's environment and flies in the 3D camera's field of view as if part of the user's real environment. Those skilled in the art will recognize from the disclosure herein that unlike the user 110 shown in FIG. 1, the whale image 118 does not actually exist in the surrounding environment. Rather, the whale image 118 is shown in FIG. 1 to represent the augmented reality experienced by the user 110 when viewing a video of the virtual image 116 of the flying whale through the screen on the handheld device 112.”) In addition, the same motivation is used as the rejection for claim 1.
Regarding claim 22, Abovitz, Mullins_371 and KAMHI teach the method of claim 1, wherein the wearable device is associated with a totem configured to be operated by the user ([0641] As illustrated in FIG. 63, the AR system may render a user interface virtual construct including a plurality of user selectable virtual elements, so at to appear in a user's field of view. The user manipulates a totem to interact with the virtual elements of the user interface virtual construct. The user, may for example, point a front of the totem at a desired one of the elements. The user may also interact with the totem, for example tapping or touching on a surface of the totem, indicating a selection of the element at which the totem is pointing or aligned. The AR system detects the orientation of the totem and the user interactions with the totem, interpreting such as a selection of the element at which the totem is pointing or aligned. The AR system the executes a corresponding action, for example opening an application, opening a submenu, or rendering a virtual room or virtual space corresponding to the selected element. [0642] The totem may replicate a remote control, for example remote controls commonly associated with televisions and media players. In some implementations, the totem may be an actual remote control for an electronic device (e.g., television, media player, media streaming box), however the AR system may not actually received any wireless communications signals from the remote control. The remote control may even not have batteries, yet still function as a totem since the AR system is relies on image that capture position, orientation and interactions with the totem (e.g., remote control).”), and
wherein the method further comprises determining a function of the totem based on the physiological parameter (see at least: [0409] of Abovitz “Thus a totem may be any object on which virtual content can be rendered, including for example a body part (e.g., hand) to which virtual content can be locked in a user experience (UX) context. In some implementations, the AR system can render virtual content so as to appear to be coming out from behind a totem, for instance appearing to emerge from behind a user's hand, and slowly wrapping at least partially around the user's hand. The AR system detects user interaction with the virtual content, for instance user finger manipulation with the virtual content which wrapped partially around the user's hand. Alternatively, the AR system may render virtual content so as to appear to emerge from a palm of the user's hand, and detection user fingertip interaction or manipulate of that virtual content. Thus, the virtual content may be locked to a reference from of a user's hand. The AR system may be responsive to various user interactions or gestures, including looking at some item of virtual content, moving hands, touching hands to themselves or to the environment, other gestures, opening and/or closing eyes, etc.” [0603] “ As illustrated in FIG. 60A, the user executes a first gesture (illustrated by double headed arrow), to open an icon based cluster user interface virtual construct (FIG. 60B). The gesture may include movement of the user's arms and/or hands or other parts of the user's body, for instance head pose or eyes. Alternatively, the user may use spoken commands to access the icon based cluster user interface virtual construct (FIG. 60B). If a more comprehensive menu is desired, the user may use a different gesture.” Where gesture may include movement of the user's arms and/or hands or other parts of the user's body, for instance head pose or eyes which is considered as physiological parameter.”; [0641] As illustrated in FIG. 63, the AR system may render a user interface virtual construct including a plurality of user selectable virtual elements, so at to appear in a user's field of view. The user manipulates a totem to interact with the virtual elements of the user interface virtual construct. The user, may for example, point a front of the totem at a desired one of the elements. The user may also interact with the totem, for example tapping or touching on a surface of the totem, indicating a selection of the element at which the totem is pointing or aligned. The AR system detects the orientation of the totem and the user interactions with the totem, interpreting such as a selection of the element at which the totem is pointing or aligned. The AR system the executes a corresponding action, for example opening an application, opening a submenu, or rendering a virtual room or virtual space corresponding to the selected element. [0642] The totem may replicate a remote control, for example remote controls commonly associated with televisions and media players. In some implementations, the totem may be an actual remote control for an electronic device (e.g., television, media player, media streaming box), however the AR system may not actually received any wireless communications signals from the remote control. The remote control may even not have batteries, yet still function as a totem since the AR system is relies on image that capture position, orientation and interactions with the totem (e.g., remote control).” Where user can interact totem by touch or tap with other gestures, opening and/or closing eyes,; [0083] of Mullins_371 “FIG. 6C is a block diagram illustrating an example of a visualization of an action on a virtual object in the device based on thought in combination with a visual gesture, according to some example embodiments. [0084] The front-facing camera 614 of the device 600 may capture and monitor a gaze of the user 601. In other words, the device 600 may be capable of tracking eye movement to determine where on the display 602, the user is looking at. In another embodiment, the front-facing camera 614 of the device 600 determines the gaze of the user 601 using a combination of eye-tracking movement and head-tracking movement.” ) In addition, the same motivation is used as the rejection for claim 1.
Regarding claim 23, Abovitz, Mullins_371 and KAMHI teach the method of claim 1, further comprising obtaining the data associated with the physiological parameter of the user using an inward-facing imaging system configured to image at least a portion of a face of the user such that the mood of the user is at least partially determined according to one or more facial expressions of the user (see at least: [0672] of Abovitz “ In some implementations, the AR system may capture images of the customers, for example via inward facing cameras carried by each customer's individual head worn component. The AR system may provide a composited virtual image to the celebrity of a crowd composed of the various customers.”; see at least [0037] of Mullins_371 The sensors 102 may include electrodes that measure electrical activity from a human. For example, the sensors 102 may include electrodes to measure EEG (electroencephalography) waves of brains, EMG (electromyography) waves of muscles, and EOG (electrooculogram) waves of eyes. The sensors 102 can be used to monitor brainwaves through EEG by detecting electrical signals about a person's level of concentration or state of mind. The sensors may be implemented, for example, by using a headset attached to a head of a user. In another example, the sensors 102 can be used to monitor facial muscles to detect facial expressions of the user. [0038] “In another example embodiment, the sensors 102 may also include: an optical sensor (e.g., a charged-coupled device (CCD)), an orientation sensor (e.g., gyroscope), and/or an audio sensor (e.g., a microphone). For example, the device 100 may include a front-facing camera for tracking eyes movement and facial expression of the user, and a rear-facing camera for capturing a picture or a video of a physical object (or another displayed virtual object). It is noted that the sensors 102 described herein are for illustration purposes and the sensors 102 are thus not limited to the one described. In another example, sensors 102 may not be physically connected to the device 100 but are instead coupled to the device 100 via wireless means such as Wi-Fi and Bluetooth®.” [0079] The device 600 generates a visualization of a three-dimensional virtual object in a display 602 of the device 600 based on outputs from sensors 610 and the visual reference 606. For example, the device 600 may determine that the user 601 is geographically located at an architectural firm. The device 600 determines from the sensors 610 that the state of mind of the user 601 is focused on a high rise building. In another embodiment, a front facing camera 614 of the device 600 may further enhance and provide additional data on the state of mind of the user 601. For example, the device 600 may obtain a live picture of the user 601 using the front facing camera 614 to determine a smile or a frown. In another example, the front facing camera 614 may be used for facial recognition to determine the identity of the user 601. The device 600 may retrieve preferences from the user 601 such as, for example, favorite colors or items. In another example, the device 600 determines, identifies, and manipulates a virtual object to be displayed in the display 602 based on a combination of the geographic location of the device 600 (e.g., office, home, restaurant, city, country), time of capture (e.g., morning, afternoon, evening, holiday, weekend) of the visual reference 606, orientation (e.g., portrait or landscape, how close) of the device 600 relative to the visual reference 606, identification of the user 601 (e.g. using facial recognition, or login information), preferences of the user 601 (e.g., favorite color, favorite type of music) social network information (e.g, number of friends, interests, proximity of friends, postings) related to the user 601, outputs from sensors 610 (e.g., EEG brain waves, EMG muscles waves, EOG eyes waves, heart rate, blood pressure), and the visual reference 606.”; [0022] of KAMHI “ FIG. 1 graphically represents an example of augmented reality according to one embodiment. In this example, a user 110 starts an AR module (not shown) on a handheld device 112 (e.g., computer tablet) and is presented with a suggested list of virtual 3D models. The list may be based on the user's location, mood, and/or user profile. For example, if the AR module determines that the user 110 is located in corn field that appears as a “sea” of corn stalks 114, and that the user 110 is in a playful mood, the list may include a virtual 3D model comprising a “flying whale” configured to playfully dive in and out of the corn stalks 114. After selecting the virtual 3D model from the list, the user 110 captures a micro video clip by positioning a world facing 3D camera of the handheld device 112 (e.g., tablet) to a view of interest (e.g., an area in and above the corn field). The AR module displays a virtual image 116 of the flying whale on a screen of the handheld device 112. The virtual image 116 of the flying whale blends into the user's environment and flies in the 3D camera's field of view as if part of the user's real environment. Those skilled in the art will recognize from the disclosure herein that unlike the user 110 shown in FIG. 1, the whale image 118 does not actually exist in the surrounding environment. Rather, the whale image 118 is shown in FIG. 1 to represent the augmented reality experienced by the user 110 when viewing a video of the virtual image 116 of the flying whale through the screen on the handheld device 112.”). In addition, the same motivation is used as the rejection for claim 1.
Regarding claim 24, Abovitz, Mullins_371 and KAMHI teach the method of claim 1, wherein the selecting first virtual content to be presented to the user from the potential virtual content includes selecting entertainment content based at least partly on the evaluation of the physiological state of the user and the current location of the user (see at least: [0601] of Abovitz “ FIGS. 60A, 60B (scenes 6002 and 6004) show a user sitting in a physical living room space, and using an AR system to experience a virtual room or virtual space in the form of a virtual entertainment or media room, the user executing gestures to interact with a user interface virtual construct, according to one illustrated embodiment. [0602] The physical living room may include one or more physical objects, for instance walls, floor, ceiling, a coffee table and sofa. As previously noted, the user may wear a head worn AR system, or head worn component of an AR system, operable to render virtual content in a field of view of the user. For example, the head worn AR system or component may render virtual objects, virtual tools and applications onto the retina of each eye of the user.[0603] As illustrated in FIG. 60A, the user executes a first gesture (illustrated by double headed arrow), to open an icon based cluster user interface virtual construct (FIG. 60B). The gesture may include movement of the user's arms and/or hands or other parts of the user's body, for instance head pose or eyes. Alternatively, the user may use spoken commands to access the icon based cluster user interface virtual construct (FIG. 60B). If a more comprehensive menu is desired, the user may use a different gesture. [0604] As illustrated in FIG. 60B, the icon based cluster user interface virtual construct provides a set of small virtual representations of a variety of different virtual rooms or spaces from which a user may select. This virtual user interface provides quick access to virtual rooms or virtual spaces via representations of the virtual rooms or virtual spaces. The small virtual representations are themselves essentially non-functional, in that they do not include functional virtual content. Thus, the small virtual representations are non-functional beyond being able to cause a rendering of a functional representation of a corresponding virtual room or space in response to selection of one of the small virtual representations. [0605] The set of small virtual representations may correspond to a set or library of virtual rooms or spaces available to the particular user. Where the set includes a relatively large number of choices, the icon based cluster user interface virtual construct may, for example, allow a user to scroll through the choice. For example, in response to a second gesture, an AR system may re-render the icon based cluster user interface virtual construct with the icons shifted in a first direction (e.g., toward user's right), with one icon falling out of a field of view (e.g., right-most icon) and a new icon entering the field of view. The new icon corresponds to a respective virtual room or virtual space that was not displayed, rendered or shown in a temporally most immediately preceding rendering of the icon based cluster user interface virtual construct. A third gesture may, for example, cause the AR system to scroll the icons in the opposite direction (e.g., toward user's left) similar to process flow diagram of FIG. 37).[0606] In response to a user selection of a virtual room or virtual space, the AR system may render virtual content associated with the virtual room or virtual space to appear in the user's field of view. The virtual content may be mapped or "glued" to the physical space. For example, the AR system may render some or all of the virtual content positioned in the user's field of view to appear as if the respective items or instances of virtual content are on various physical surfaces in the physical space, for instance walls, tables, etc. Also for example, the AR system may render some or all of the virtual content positioned in the user's field of view to appear as if the respective items or instances of virtual content are floating in the physical space, for instance within reach of the user.”; [0079] of Mullins_371; [0022] of KAMHI “FIG. 1 graphically represents an example of augmented reality according to one embodiment. In this example, a user 110 starts an AR module (not shown) on a handheld device 112 (e.g., computer tablet) and is presented with a suggested list of virtual 3D models. The list may be based on the user's location, mood, and/or user profile. For example, if the AR module determines that the user 110 is located in corn field that appears as a “sea” of corn stalks 114, and that the user 110 is in a playful mood, the list may include a virtual 3D model comprising a “flying whale” configured to playfully dive in and out of the corn stalks 114. After selecting the virtual 3D model from the list, the user 110 captures a micro video clip by positioning a world facing 3D camera of the handheld device 112 (e.g., tablet) to a view of interest (e.g., an area in and above the corn field). The AR module displays a virtual image 116 of the flying whale on a screen of the handheld device 112. The virtual image 116 of the flying whale blends into the user's environment and flies in the 3D camera's field of view as if part of the user's real environment. Those skilled in the art will recognize from the disclosure herein that unlike the user 110 shown in FIG. 1, the whale image 118 does not actually exist in the surrounding environment. Rather, the whale image 118 is shown in FIG. 1 to represent the augmented reality experienced by the user 110 when viewing a video of the virtual image 116 of the flying whale through the screen on the handheld device 112.”) In addition, the same motivation is used as the rejection for claim 1.
2. Claims 2 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Abovitz et al.,IDS, U.S Patent Application Publication No.20150016777(“Abovitz”) in view of Mullins et al, U.S Patent Application Publication No.20160246371 (“Mullins_371”) further in view of KAMHI et al., U.S Patent Application Publication No. 20160180590 (“KAMHI”) further in view of Petersen et al, U.S Patent Application Publication No.20150371516 (“Petersen”)
Regarding claim 2, Abovitz, Mullins_371 and KAMHI teach the method of claim 1, wherein the physiological parameter of the user (see at least: [0335] of Abovitz Referring to process flow diagram 3000 of FIG. 30, on a basic level, the AR system is configured to receive input (e.g., visual input 2202 from the user's wearable system, input from room cameras 2204, sensory input 2206 in the form of various sensors in the system, gestures, totems, eye tracking etc.) from one or more AR systems. The AR systems may constitute one or more user wearable systems, and/or stationary room systems (room cameras, etc). The wearable AR systems not only provide images from the FOV cameras, they may also be equipped with various sensors (e.g., accelerometers, temperature sensors, movement sensors, depth sensors, GPS, etc.), as discussed above, to determine the location, and various other attributes of the environment of the user. Of course, this information may further be supplemented with information from stationary cameras mentioned previously that may provide images and/or various cues from a different point of view. It should be appreciated that image data may be reduced to a set of points, as explained above.”; [0025] In another example embodiment, the device also includes an EEG brain wave sensor coupled to a head of the user to generate brain wave data, a heart rate sensor to generate heart rate data of the user, an eye tracking sensor to determine a gaze of the user on the display, a location sensor to determine a geographic location of the device, and a social network module configured to access social network information associated with the user. The electric wave application identifies the state of mind of the user based on the brain wave data, heart rate data, the gaze of the user, the geographic location of the device, and social network information associated with the user.”) In addition, the same motivation is used as the rejection for claim 1. Abovitz, Mullins_371 and KAMHI are understood to be silent on the remaining limitations of claim 2.
In the same field of endeavor, Petersen teaches wherein the physiological parameter of the user is a galvanic skin response ([0016] As described herein, one or more computing devices may determine when to output information, determine the content of the information, or otherwise manage how information is delivered to a user based on one or more physiological parameters detected from the user. In this manner, the computing device may be configured to determine a condition of the user and output information relevant to that determined condition. For example, a computing device may withhold at least some notification information based on a received physiological parameter. The computing device may be configured to receive an indication of a physiological parameter representative of a physiological condition of a user. The physiological parameter may be representative of a galvanic skin response (GSR), a pulse rate, a breathing rate, or an electrogram of the user. In some examples, one or more sensors of a wearable computing device (e.g., a watch, a wristband, a smartwatch, a chest strap, smart eye glasses, or any other such devices) may generate a signal indicative of the physiological parameter. [0027] As described herein, computing device 10 may be configured to manage or control the delivery of information to a user (e.g., a user of computing device 10). For example, computing device 10 may be configured to determine whether to output some or all of notification information received from information server system 60 in a request to output the notification information. Computing device 10 may receive physiological parameters regarding the physical condition of the user of computing device 10. One or more sensors may generate physiological parameters, or values of physiological parameters, from detectable physiological signals from the user. For example, physiological parameters may be representative of a galvanic skin response, a breathing rate, a pulse rate, a temperature, an electrogram, or any other detectable signal or physiological state. Computing device 10 may include one or more of the sensors (e.g., within housing 18 of computing device 10 or attached to housing 18) or be in wired or wireless communication with one or more remote sensors associated with the user. [0059] One or more sensors 14 may include one or more physiological sensors for obtaining physiological parameter information associated with a user of computing device 10. For example, one or more sensors 14 may include a heart monitor sensor, a temperature sensor, a galvanic skin response sensor, an accelerometer, a gyroscope, a pressure sensor, a blood pressure sensor, and/or any other sensor for measuring a physiological parameter that computing device 10 may use for determining a physiological condition of a user. In some examples, input devices 42 may include one or more of sensors 14.”)
Therefore, it would have been obvious to one of ordinary skill in art before the effective filling date to the claimed invention to modify display virtual content associated with physical surface of Abovitz, Mullins_371 and KAMHI with including a galvanic skin response sensor for measuring a physiological parameter as seen in Petersen because this modification would determine a physiological condition of a user ([0059] of Petersen)
Thus, the combination of Abovitz, Mullins_371, KAMHI and Petersen teaches wherein the physiological parameter of the user is a galvanic skin response.
Regarding claim 17, Abovitz, Mullins_371, KAMHI teach the wearable device of claim 16, Remaining limitations of claim 17 is similar scope to claim 2 and therefore rejected under the same rational.
3. Claims 25 is rejected under 35 U.S.C. 103 as being unpatentable over Abovitz et al. ,IDS, U.S Patent Application Publication No.20150016777(“Abovitz”) in view of Mullins et al, U.S Patent Application Publication No.20160246371 (“Mullins_371”) further in view of KAMHI et al., U.S Patent Application Publication No. 20160180590 (“KAMHI”) further in view of Cruz Hernandez, U.S Patent Application Publication No. 20100123588 (“Cruz Hernandez”) further in view of ALLEN et al. U.S Patent Application Publication No. 20100153868 (“ALLEN”)
Regarding claim 25, Abovitz, Mullins_371 and KAMHI teach the method of claim 1,
wherein the user is a first user ([0681] of Abovitz “ In particular, FIG. 68A shows a surgeon and surgical team, including a virtually rendered consulting or visiting surgeon, conducting a pre-operative planning session for an upcoming mitral valve replacement procedure. Each of the health care providers is wearing a respective individual AR system., and the method further includes obtaining, using respective physiological sensors, respective physiological parameters of additional users interacting with the first user”), and the method further includes obtaining, using respective physiological sensors, respective physiological parameters of additional users (see at least: [0335] of Abovitz Referring to process flow diagram 3000 of FIG. 30, on a basic level, the AR system is configured to receive input (e.g., visual input 2202 from the user's wearable system, input from room cameras 2204, sensory input 2206 in the form of various sensors in the system, gestures, totems, eye tracking etc.) from one or more AR systems. The AR systems may constitute one or more user wearable systems, and/or stationary room systems (room cameras, etc). The wearable AR systems not only provide images from the FOV cameras, they may also be equipped with various sensors (e.g., accelerometers, temperature sensors, movement sensors, depth sensors, GPS, etc.), as discussed above, to determine the location, and various other attributes of the environment of the user. Of course, this information may further be supplemented with information from stationary cameras mentioned previously that may provide images and/or various cues from a different point of view. It should be appreciated that image data may be reduced to a set of points, as explained above.”; [0672] In some implementations, the AR system may capture images of the customers, for example via inward facing cameras carried by each customer's individual head worn component. The AR system may provide a composited virtual image to the celebrity of a crowd composed of the various customers.”; [0684] “The AR system may also render patient information. For instance, the AR system may render some patient information (e.g., identification information) so as to appear on a surface of a physical table. Also for instance, the AR system may render other patient information (e.g., medical images, vital signs, charts) so as to appear on a surface of one or more physical walls. Similar to the process flow of FIG. 55, the AR system may detect and recognize input (e.g, here the users may explicitly request to see virtual representation of the pre-mapped anatomy of the heart). Here, based on input, the AR system may retrieve the data from the cloud server, and transmit it back to the user's devices. The system also uses the map coordinates of the room to display the virtual content in the center of the room so that it can be viewed by multiple users sitting around the table.”; [0049] of Mullins_371 “ The sensor module 202 captures outputs from the sensors 102. For example, the sensor module 202 may capture electric waves generated by a brain of the user by using EEG electrodes positioned on the scalp of the user. As previously described, the sensor module 202 may also capture outputs from other types of sensors such as a heart rate monitor or a facial muscle monitor to further supplement outputs from the EEG electrodes. In another embodiment, the sensors 102 may include a camera to detect the gaze of the user and determine where the user is look on the display 104. [0050] The initialization module 208 enables the electric wave application 110 to initialize and calibrate outputs from sensors 102 based a particular user. The user may be asked to relax while the initialization module 208 captures a sample of outputs from the sensors while the user in a relaxed state of mind. The sample outputs may then be used as a baseline or a reference for the user. [0051] The intensity module 204 measures the intensity of one or more sensors. For example, the intensity module 204 may measure the intensity of electrical signals of Alpha waves in combination with the heart rate of a user to determine the user's relaxed state of mind. In another embodiment, the intensity may be based on a statistical computation (e.g., average or median) of one or more outputs from selected sensors.”) In addition, the same motivation is used as the rejection for claim 1. Abovitz, Mullins_371 and KAMHI are understood to be silent on the remaining limitation of claim 25.
In the same field of endeavor, Cuz Hernandez teaches wherein the user is a first user and the method further includes obtaining, using respective physiological sensors, respective physiological parameter of additional user interacting with the first user ([0020] Sensing device 114, in one embodiment, includes multiple sensors configured to sense and collect mood information relating to a user. For example, sensing device 114 employs sensors to detect and/or collect mood information via various modalities or user's mood states, which include, but not limited to, user's facial expressions, voice pitches, and/or user's biometrics. The biometrics, which is a subset of user's mood states, further include body temperature, body humidity or perspiration, heart pulse or rate, breathing rhythms, body posture, hand gestures or movements, and so forth. After obtaining the mood information, a digital processing unit, which could be onboard with portable device 102 or 106, identifies user's current mood and/or psychological condition(s) based on the collected mood information. The information relating to user's mood is subsequently forwarded to interested parties, who can be the user himself or herself or person(s) who is interacting with the user. For example, when two persons are talking over cellular phones, either party can sense the other person's mood over the mood-based haptic mechanism.”; [0030] Haptic system, in one embodiment, can include multiple units wherein some of the units may be located in the chest, wrist, foot, and/or the like to sense user's mood. Haptic generator 128, for example, is capable of generating haptic cues or haptic warning signals at different levels of intensities for different levels of mood swings. For example, haptic generator 128 generates a minor haptic cue when the user is slightly unhappy, and generates an intensified haptic cue when the user is mad. It should be noted that using tactile feedback to indicate the user's physiological conditions can be a subtle, discreet, and non-intrusive communication method.”) Therefore, it would have been obvious to one of ordinary skill in art before the effective filling date to the claimed invention to modify display virtual content associated with physical surface of Abovitz, Mullins_371 and KAMHI with detecting person’s mood when two persons are interacting of Cuz Hernandez because this modification would sense the other person's mood over the mood-based haptic mechanism ([0020] of Cuz Hernandez) Abovitz, Mullins_371, KAMHI and Cuz Hernandez are understood to be silent on the remaining limitations of claim 25.
In the same field of endeavor, ALLEN teaches wherein the user is a first user and the method further includes obtaining, respective physiological parameter of additional user interacting with the first user ([0052] A financial transaction agent may be used to obtain information about a user by querying financial company records for recent transactions that may be used to infer a users state of mind or health. For example, the financial transaction agent may detect that user is a buying a book, paying for a plane ticket, or purchasing a piece of health related software, etc. One or more of these detected events may be sent to the Emotion Engine as an input and used to determine one or more possible states of mind, emotions, etc”) and determining an overall physiological state of the first user and the additional users based on the physiological state of the first user and the physiological parameters of the additional users ([0073] Moreover, in embodiments, an aggregate mood for a group of avatars may be computed. For example, an aggregate mood may be computed for a group of avatars located on an island. The aggregate mood may be detected by one or more of surveys, typing speed, use of curse words in communications, number of sick days taken during a specified duration, or any number of additional inputs described above. The aggregated mood may be represented to one or more users via a visual or numerical scale. For example, a dark blue hue may indicate a poor mood on the island. Based on the aggregated mood, a number of environmental factors, such as the weather, can be changed to positively or negatively affect the mood. Thus, the weather on an island having avatar's with an aggregate bad mood may be altered to provide sun and bright skies in order to improve the aggregate mood.”); and
wherein the selecting of the first virtual content is at least partly based on the overall physiological state of the first user and the additional users ([0073] Moreover, in embodiments, an aggregate mood for a group of avatars may be computed. For example, an aggregate mood may be computed for a group of avatars located on an island. The aggregate mood may be detected by one or more of surveys, typing speed, use of curse words in communications, number of sick days taken during a specified duration, or any number of additional inputs described above. The aggregated mood may be represented to one or more users via a visual or numerical scale. For example, a dark blue hue may indicate a poor mood on the island. Based on the aggregated mood, a number of environmental factors, such as the weather, can be changed to positively or negatively affect the mood. Thus, the weather on an island having avatar's with an aggregate bad mood may be altered to provide sun and bright skies in order to improve the aggregate mood.[0074] In embodiments, any number of states such as emotional, financial, and/or health, etc., may be aggregated and reflected in the characteristics of a building or landscape, which are associated with a plurality of avatars. For example, if 50 avatars are on an island or in a building, and if their average real-world or virtual-world income is above a threshold, this may be reflected in attributes in the virtual world. Similarly, if the avatars are judged to have an average emotional state, this may also be reflected in the depiction of the associated landscape or building. Such depictions may also reflect states over a period of time, for example, the current state or the state over the past month.”)
Therefore, it would have been obvious to one of ordinary skill in art before the effective filling date to the claimed invention to modify display virtual content associated with physical surface of Abovitz, Mullins_371 and KAMHI and detecting person’s mood when two persons are interacting of Cuz Hernandez with determining aggregated mood of group of users as seen in ALLEN because this modification would change a number of environmental factors, such as the weather, to positively or negatively affect the mood based on the aggregated mood ([0073] of ALLEN).
Thus, the combination of Abovitz, Mullins_371, KAMHI, Cuz Hernandez and ALLEN teaches wherein the user is a first user and the method further includes obtaining, using respective physiological sensors, respective physiological parameter of additional user interacting with the first user; and determining an overall physiological state of the first user and the additional users based on the physiological state of the first user and the physiological parameters of the additional users, and wherein the selecting of the first virtual content is at least partly based on the overall physiological state of the first user and the additional users.
Contact
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SARAH LE whose telephone number is (571)270-7842. The examiner can normally be reached Monday: 8AM-4:30PM EST, Tuesday: 8 AM-3:30PM EST, Wednesday: 8AM-2:30PM EST, Thursday and Friday off.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kent Chang can be reached at (571) 272-7667. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SARAH LE/Primary Examiner, Art Unit 2614