DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . Claims 1-20 are pending under this Office action.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-6, 8-14, and 16-20 are rejected under 35 U.S.C. 103 as being unpatentable over Swaminathan, etc. (US 20140168056 A1) in view of Jiang, etc. (US 20120062702 A1).
Regarding claim 1, Swaminathan teaches that a computer-implemented method (See Swaminathan: Fig. 1, and [0041], “FIG. 1 illustrates an exemplary computer system incorporating parts of the device employed in practicing embodiments of the invention. A computer system as illustrated in FIG. 1 may be incorporated as part of the computerized devices described below. For example, computer system 100 can represent some of the components of a mobile device or a smart phone. A mobile device may be any computing device with an input sensory unit like a camera and may also include a display unit. Examples of a mobile device include but are not limited to video game consoles, tablets, smart phones and any other hand-held devices. FIG. 1 provides a schematic illustration of one embodiment of a computer system 100 that can perform the methods provided by various other embodiments, as described herein, and/or can function as the host computer system, a remote kiosk/terminal, a point-of-sale device, a mobile device, a set-top box and/or a computer system. FIG. 1 is meant only to provide a generalized illustration of various components, any or all of which may be utilized as appropriate. FIG. 1, therefore, broadly illustrates how individual system elements may be implemented in a relatively separated or relatively more integrated manner”) comprising:
determining a location within a virtual or augmented reality (VAR) scene at which to provide dynamic content (See Swaminathan: Figs. 8-13, and [0089], “At stage 1302 the device 402 may obtain an image with the front-facing camera. In an embodiment, image 702 may be a street view that is presented on the display 408 in a camera view or AR view mode. As depicted in FIG. 8, the target image may be a file displayed in a browser or other application (e.g., Google Streetview, Yelp Monocle). The device 402 may exchange location information with a location based services system. For example, the location information can be based on a GPS position, WiFi based position, or other positioning techniques as known in the art. The location information may also include an approximate heading as derived from onboard GPS chips and other internal sensors. A location based services system can provide augmented reality information to the device 402 based on the location information. For example, referring to FIG. 7, the location based services system can provide object tags 704 which are associated with locations depicted in the image 702”);
providing the VAR scene for display on a viewer device (See Swaminathan: Figs. 1-4, and [0059], “Referring to FIG. 4, with further reference to FIGS. 1 and 2, an exemplary illustration of using eye gaze tracking to identify an area of interest on an image captured with a front-facing camera on a device 402 is shown. In an embodiment, the device 402 is a mobile device 100. The device 402 includes a front-facing camera (not shown), a back-facing camera 406, and a display 408. The image in the display 408 includes a road sign 404, a background light enhancing region 410, a graphical representation of an area of interest 412, and an area of interest 414. In operation, a user can direct the front-facing camera toward a real time scene such as a road sign 404. The image can be presented on the display 408 in a camera view or an AR view mode. The image can also be stored in memory 106. The back-facing camera 406 may be used to determine an area of interest 414 based on an eye gaze tracking algorithm (e.g., FIG. 3). In an embodiment, the display 408 can activate a background light enhancing region 410 to help illuminate the user's eyes. For example, a region of the display 408 can be set to a bright white color to provide more light towards the user's face and eyes. In an embodiment, the region 410 can form a frame around the image. Other shapes and patterns such as circles or bright dots, or different shapes in the corners of the display 408 may be used. The shape of the enhancing region 410 can be used to help the camera 406 detect a key feature at stage 310 in an eye gazing process 300”);
detecting, based on data from one or more sensors of the viewer device (See Swaminathan: Figs. 3-4, and [0058], “Referring to FIG. 3, an exemplary prior art process flow 300 for performing eye gaze tracing with a back-facing camera on a mobile device is shown. The process 300, however, is exemplary only and not limiting. The process 300 may be altered, e.g., by having stages added, removed, or rearranged. Other process for receiving eye gaze tracking information may be used. At stage 302, the mobile device 200 can utilize the back-facing camera 206 to provide a video frame to one or more processors 104. The process can include an optional step 304 of utilizing a face detection algorithm to identify the face of the user to pass the coordinates of the eyes to an eye detection algorithm 306. The eye gaze tracking information can be based on the relative position of a user's iris. For example, the eye detection algorithm can pass the coordinates of the eyes to an iris detection algorithm at stage 308. The coordinates of the iris can be passed to a pre-processing algorithm to extract key features from the eye at stage 310. For example, the size and point of the iris can be mapped and a segment of fixed size can be selected. The brightness of each pixel within the segment can be used and input value to a learning algorithm at stage 312. By means of an example, a neural network may be used for the learning algorithm. In an example, the neural network is a basic two-layer network with a symmetric sigmoid activation function. Additional layers can be used to increase the accuracy of the neural network. Two output neurons can be used for the (x,y) screen coordinates at stage 314. The screen coordinates can be the basis for an area of interest which can be used by an application running on the device 100”; and [0059], “Referring to FIG. 4, with further reference to FIGS. 1 and 2, an exemplary illustration of using eye gaze tracking to identify an area of interest on an image captured with a front-facing camera on a device 402 is shown. In an embodiment, the device 402 is a mobile device 100. The device 402 includes a front-facing camera (not shown), a back-facing camera 406, and a display 408. The image in the display 408 includes a road sign 404, a background light enhancing region 410, a graphical representation of an area of interest 412, and an area of interest 414. In operation, a user can direct the front-facing camera toward a real time scene such as a road sign 404. The image can be presented on the display 408 in a camera view or an AR view mode. The image can also be stored in memory 106. The back-facing camera 406 may be used to determine an area of interest 414 based on an eye gaze tracking algorithm (e.g., FIG. 3). In an embodiment, the display 408 can activate a background light enhancing region 410 to help illuminate the user's eyes. For example, a region of the display 408 can be set to a bright white color to provide more light towards the user's face and eyes. In an embodiment, the region 410 can form a frame around the image. Other shapes and patterns such as circles or bright dots, or different shapes in the corners of the display 408 may be used. The shape of the enhancing region 410 can be used to help the camera 406 detect a key feature at stage 310 in an eye gazing process 300”. Note that using back-facing/eye-tracking camera sensors to determine gaze and region of interest is mapped to the projection vector corresponding to the scene location), that a viewer projection vector corresponds with the location within the VAR scene; and
providing, for display, the dynamic content within the VAR scene (See Swaminathan: Fig. 1, and [0003], “Mobile computing devices (mobile devices) provide users with access to a variety of information via wireless communication systems. As an example, mobile devices enabled for use with wireless cellular networks and/or wireless local area networks such as Wi-Fi or WiMAX provide users with access to vast information resources of the Internet. Mobile devices may also enable users to explore augmented reality environments which provide a real-time view of a target object merged with, or augmented by, computer generated graphical content. For example, cameras residing on-board a mobile device may be used in conjunction with a graphical display to present a user with supplemental information relating to targets of interest that are captured in a camera view of the mobile device. Such supplemental information may form an information layer that overlays real-world objects that are captured in a camera view of the mobile device. This functionality is generally referred to as Augmented Reality (AR) view mode. In an AR view, objects captured in the camera view can be transitioned to align with the display screen to allow a user easier access to the information layer”; and [0060], “In an embodiment, the area of interest 414 can be represented on the display 408 by a graphical representation such as one or more circles 412 to provide feedback to the user. The diameter of the circles 412 can be a function of the amount of time the user's gaze lingers in an area (e.g., the more time, the larger the diameter). Multiple circles can be used to create a tracking effect as the user's gaze moves across the display 408. The circles are exemplary only and not a limitation as other shapes or indications may be used”. Note that the overlay and the circle 412 may be mapped to the dynamic content) based on the viewer projection vector corresponding with the location.
However, Swaminathan fails to explicitly disclose that that a viewer projection vector corresponds with the location within the VAR scene; and (providing, for display, the dynamic content within the VAR scene) based on the viewer projection vector corresponding with the location.
However, Jiang teaches that that a viewer projection vector corresponds with the location within the VAR scene (See Jiang: Figs. 1-5, and [0022], “FIG. 5 illustrates a 3D point Q on a plane .pi. and the 2D projection q, q' of the 3D point Q on two respective images I, I' with different views of the plane .pi. to illustrate determining the pose of the master device 110A using the following notation”; and [0035], “For a 3D point Q on the plane .pi., its 2D projection q and the center of the camera 114, illustrated in FIG. 5 as point O, forms a ray. After intersecting the ray with the plane .pi., the coordinate for the 3D point Q can be determined. The plurality of reference points, correspond to a plurality of 3D points, which form a 3D plane. Generally, there are two solutions for homography decomposition. To select the correct solution, the two resulting 3D planes are stored and used to estimate the master device 110A pose in subsequent frames. When the average projection error for one plane is greater than the average projection error for the other plane, e.g., one is 1.2 times greater; the plane having the larger projection error is eliminated. If the initial image is roughly a front view of the planar object 102, the plane normal n may be used to select the correct 3D plane. After the correct 3D plane is selected, a world coordinate frame is defined to align the z-axis with the plane normal n with the origin on the plane. Thus, with the 3D-2D homography H determined, the pose of the master device 110A with respect to the object 102 is determined based on the rotation matrix R and translation vector t as discussed above with reference to equation 6. If desired, other pose determination techniques may be used”); and
(providing, for display, the dynamic content within the VAR scene) based on the viewer projection vector corresponding with the location (See Jiang: Fig. 1, and [0002], “An augmented reality system can insert virtual objects in a user's view of the real world. One key requirement of a successful augmented reality system is a tracking system that can estimate the user's position and orientation (pose) accurately relative to a reference. Otherwise, the virtual objects will appear at the wrong location or float around the environment. In a multi-user augmented reality system, the virtual objects need to appear at the same location in the environment from each user's unique perspective. Thus, each user's unique pose with respect to the environment needs to be estimated relative to the same reference”; [0019], “FIG. 2 is a flow chart describing the process of performing AR with multi-users without a previously acquired common reference. The master device 110A captures two or more images of an object 102 with the back facing camera 114 (202). It should be understood that as used herein, a captured image may be a still image or a video frame. The two or more images of the object 102 are captured by the master device 110A at different viewpoints, i.e., poses with respect to the object 102, or by different mobile platforms 110A and 110B having different viewpoints. Using the plurality of captured images, the pose (position and orientation) of the master device 110A with respect to the object 102 is determined (204). An image of the object 102, which may be one of the initial images captured by the master device 110A or a new or different image of the object 102, is then warped based on the orientation of the master device 110A to produce a reference image 104 as a front view of the object 102 (206) as illustrated by arrow 106 in FIG. 3. The reference image 104 is used as the common reference image from which each mobile platform 110 may determine its unique pose with respect to the object for tracking to perform multi-user AR (208)”; and [0042], “If desired, the original reference image 104 may be extended and the extended reference image may be distributed to the other mobile platforms. During the initialization process, the cameras of participating mobile platforms are pointed to capture images from the same part of the object 102. The master device 110A captures images from different perspectives and using the images from different perspectives, the reference image is generated, which can then be transmitted to the other users. Each mobile platform continues to capture images of the object 102 and uses the reference image to estimate the pose for a current image. If a significant part of the current image is not visible in the reference image, the reference image and current image may be merged to generate a new reference image, which can be transmitted to the other mobile platforms”. Note that inserting the virtual objects into the user’s view at the determined locations in the user’s view of the real/augmented environment, and using the reference images and poses of the master device to ensure stable dynamic content display, is mapped to the “(providing, for display, the dynamic content within the VAR scene) based on the viewer projection vector corresponding with the location”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was effectively filed to modify Swaminathan to have that a viewer projection vector corresponds with the location within the VAR scene; and (providing, for display, the dynamic content within the VAR scene) based on the viewer projection vector corresponding with the location as taught by Jiang in order to enable storing and using two resulting three-dimensional (3D) planes to estimate the master device pose in subsequent frames (See Jiang: Fig. 1, and [0035], “For a 3D point Q on the plane .pi., its 2D projection q and the center of the camera 114, illustrated in FIG. 5 as point O, forms a ray. After intersecting the ray with the plane .pi., the coordinate for the 3D point Q can be determined. The plurality of reference points, correspond to a plurality of 3D points, which form a 3D plane. Generally, there are two solutions for homography decomposition. To select the correct solution, the two resulting 3D planes are stored and used to estimate the master device 110A pose in subsequent frames. When the average projection error for one plane is greater than the average projection error for the other plane, e.g., one is 1.2 times greater; the plane having the larger projection error is eliminated. If the initial image is roughly a front view of the planar object 102, the plane normal n may be used to select the correct 3D plane. After the correct 3D plane is selected, a world coordinate frame is defined to align the z-axis with the plane normal n with the origin on the plane. Thus, with the 3D-2D homography H determined, the pose of the master device 110A with respect to the object 102 is determined based on the rotation matrix R and translation vector t as discussed above with reference to equation 6. If desired, other pose determination techniques may be used”). Swaminathan teaches a method and system that may generate and display scene images to the user based on the region of interest gazed by the user’s eye with gaze direction detection algorithm; while Jiang teaches a multiple-user augmented reality (AR) system and method that may determine the master device pose, generate reference images for the multiple-user environment, and insert (provide dynamic scene for the user) virtual object into scene at the detected location of the users at scene environment. Therefore, it is obvious to one of ordinary skill in the art to modify Swaminathan by Jiang to provide dynamic content within the scene based on the detected location and projection direction (gaze direction or orientation. The motivation to modify Swaminathan by Jiang is “Use of known technique to improve similar devices (methods, or products) in the same way”.
Regarding claim 2, Swaminathan and Jiang teach all the features with respect to claim 1 as outlined above. Further, Swaminathan teaches that the computer-implemented method of claim 1, wherein detecting that the viewer projection vector corresponds with the location within the VAR scene comprises:
determining a position of a user's eyes with respect to the VAR scene (See Swaminathan: Figs. 1-4, and [0040], “A presentation region for a media content item may be deployed to an augmented reality environment by a user for the purpose of defining a location (e.g., a position and/or orientation) where the media content item is presented within that augmented reality environment. An activation region for a media content item may be deployed within an augmented reality environment by a user for the purpose of defining conditions for how and/or when the presentation of the media content item at an associated presentation region may be enabled and/or initiated responsive to user eye gaze location and/or estimated mobile device position and/or orientation. An activation region can be within a user's eye gaze to initiate a transition from the augmented reality environment to a non-augmented reality view (i.e., an application). The transition can include stages based on the time the user's eye gaze lingers on the activation region”; and [0058], “Referring to FIG. 3, an exemplary prior art process flow 300 for performing eye gaze tracing with a back-facing camera on a mobile device is shown. The process 300, however, is exemplary only and not limiting. The process 300 may be altered, e.g., by having stages added, removed, or rearranged. Other process for receiving eye gaze tracking information may be used. At stage 302, the mobile device 200 can utilize the back-facing camera 206 to provide a video frame to one or more processors 104. The process can include an optional step 304 of utilizing a face detection algorithm to identify the face of the user to pass the coordinates of the eyes to an eye detection algorithm 306. The eye gaze tracking information can be based on the relative position of a user's iris. For example, the eye detection algorithm can pass the coordinates of the eyes to an iris detection algorithm at stage 308. The coordinates of the iris can be passed to a pre-processing algorithm to extract key features from the eye at stage 310. For example, the size and point of the iris can be mapped and a segment of fixed size can be selected. The brightness of each pixel within the segment can be used and input value to a learning algorithm at stage 312. By means of an example, a neural network may be used for the learning algorithm. In an example, the neural network is a basic two-layer network with a symmetric sigmoid activation function. Additional layers can be used to increase the accuracy of the neural network. Two output neurons can be used for the (x,y) screen coordinates at stage 314. The screen coordinates can be the basis for an area of interest which can be used by an application running on the device 100”); and
determining the viewer projection vector corresponds with the location within the VAR scene based at least in part on the position of the user's eyes with respect to the VAR scene (See Swaminathan: Figs. 1-4, and [0011], “An example of a method according to the disclosure includes displaying on a mobile device a first image captured by a first camera of the mobile device, determining a gaze of a user of the mobile device based on a second image captured by a second camera of the mobile device, determining an area of interest within the first image based on the determined gaze, and performing, at the mobile device, one or more actions associated with an augmented reality function based at least in part on the determined area of interest”; and [0060], “In an embodiment, the area of interest 414 can be represented on the display 408 by a graphical representation such as one or more circles 412 to provide feedback to the user. The diameter of the circles 412 can be a function of the amount of time the user's gaze lingers in an area (e.g., the more time, the larger the diameter). Multiple circles can be used to create a tracking effect as the user's gaze moves across the display 408. The circles are exemplary only and not a limitation as other shapes or indications may be used”. Note that the area of interest and the gaze direction from the eye location to the scene location and checking the gaze lingering and overlap, is mapped to “determining the viewer projection vector corresponds with the location within the VAR scene based at least in part on the position of the user's eyes with respect to the VAR scene”).
Regarding claim 3, Swaminathan and Jiang teach all the features with respect to claim 1 as outlined above. Further, Swaminathan teaches that the computer-implemented method of claim 1, wherein detecting that the viewer projection vector corresponds with the location within the VAR scene comprises:
determining an orientation of the viewer device (See Swaminathan: Figs. 1-4, and [0040], “A presentation region for a media content item may be deployed to an augmented reality environment by a user for the purpose of defining a location (e.g., a position and/or orientation) where the media content item is presented within that augmented reality environment. An activation region for a media content item may be deployed within an augmented reality environment by a user for the purpose of defining conditions for how and/or when the presentation of the media content item at an associated presentation region may be enabled and/or initiated responsive to user eye gaze location and/or estimated mobile device position and/or orientation. An activation region can be within a user's eye gaze to initiate a transition from the augmented reality environment to a non-augmented reality view (i.e., an application). The transition can include stages based on the time the user's eye gaze lingers on the activation region”; and [0058], “Referring to FIG. 3, an exemplary prior art process flow 300 for performing eye gaze tracing with a back-facing camera on a mobile device is shown. The process 300, however, is exemplary only and not limiting. The process 300 may be altered, e.g., by having stages added, removed, or rearranged. Other process for receiving eye gaze tracking information may be used. At stage 302, the mobile device 200 can utilize the back-facing camera 206 to provide a video frame to one or more processors 104. The process can include an optional step 304 of utilizing a face detection algorithm to identify the face of the user to pass the coordinates of the eyes to an eye detection algorithm 306. The eye gaze tracking information can be based on the relative position of a user's iris. For example, the eye detection algorithm can pass the coordinates of the eyes to an iris detection algorithm at stage 308. The coordinates of the iris can be passed to a pre-processing algorithm to extract key features from the eye at stage 310. For example, the size and point of the iris can be mapped and a segment of fixed size can be selected. The brightness of each pixel within the segment can be used and input value to a learning algorithm at stage 312. By means of an example, a neural network may be used for the learning algorithm. In an example, the neural network is a basic two-layer network with a symmetric sigmoid activation function. Additional layers can be used to increase the accuracy of the neural network. Two output neurons can be used for the (x,y) screen coordinates at stage 314. The screen coordinates can be the basis for an area of interest which can be used by an application running on the device 100”); and
determining the viewer projection vector corresponds with the location within the VAR scene based at least in part on the orientation of the viewer device (See Swaminathan: Figs. 1-6, and [0011], “An example of a method according to the disclosure includes displaying on a mobile device a first image captured by a first camera of the mobile device, determining a gaze of a user of the mobile device based on a second image captured by a second camera of the mobile device, determining an area of interest within the first image based on the determined gaze, and performing, at the mobile device, one or more actions associated with an augmented reality function based at least in part on the determined area of interest”; [0060], “In an embodiment, the area of interest 414 can be represented on the display 408 by a graphical representation such as one or more circles 412 to provide feedback to the user. The diameter of the circles 412 can be a function of the amount of time the user's gaze lingers in an area (e.g., the more time, the larger the diameter). Multiple circles can be used to create a tracking effect as the user's gaze moves across the display 408. The circles are exemplary only and not a limitation as other shapes or indications may be used”; and [0063], “Referring to FIG. 6A, with further reference to FIG. 5, an exemplary illustration of the image 502 with an AR object tag 504-1 displayed as a function of the location of the user's gaze is shown. FIG. 6A continues the example of FIG. 5 with the incorporation of the user's eye gaze information. The display 408 includes the image 502, a single object tag 504-1, an image segment including the Empire State Building 505, and an area of interest 506 determined by processing the user's eye gaze. In contrast to the several object tags 504 shown in FIG. 5, the single object tag 504-1 is shown based on its proximity to the area of interest 506. The device 402 may be configured to utilize the back-facing camera 406 and an eye gaze tracking algorithm (i.e., stored in memory 114) to identify the area of interest 506 based on the location of the user's gaze. The several object tags 504 may remain hidden until the area of interest 506 passes over or near a position that is associated with an AR target object. In this example, as depicted in FIG. 6A, the "Empire State Building" text is an object tag and can appear as the user's gaze passes over or near a segment of the image containing the Empire State Building. The image segment may be highlighted with a boundary line or other graphical enhancement (e.g., brightened, color change, raised) to indicate that augmented reality information is available. In the FIG. 6A, however, such highlighting is not illustrated, nor is a segment line surrounding the Empire State Building. In an embodiment, the location of the object tag maintains a position that is on or near the associated image segment such that the object tag will move if the image moves (e.g., when the orientation of the camera changes). The distance between the area of interest 506 and the object tag 504-1 may be based on a Cartesian coordinate system (e.g., pixels on the display 408). An image segmentation and recognition process may be used to recognize an AR target object and then make the association to one or more object tags. In an embodiment, the object tag 504-1 may indicate that additional augmented reality information is available. In the example of FIG. 6A, if the user prolongs their gaze (i.e., lingers) on the segment of the image containing the Empire State Building, or the "Empire State Building" text (i.e., the object tag), additional augmented reality information may be presented to the user. As previously described, the augmented reality information can be, but is not limited to, additional information which appears on the screen, such as media files, hyperlinks, GUI objects, interactive icons, executing one or more applications, or other Augmented Reality feature as known in the art”. Note that the area of interest and the gaze direction from the eye location to the scene location and checking the gaze lingering and overlap, is mapped to “determining the viewer projection vector corresponds with the location within the VAR scene based at least in part on the position of the user's eyes with respect to the VAR scene”).
Regarding claim 4, Swaminathan and Jiang teach all the features with respect to claim 1 as outlined above. Further, Swaminathan teaches that the computer-implemented method of claim 1, wherein providing the dynamic content within the VAR scene comprises providing the dynamic content over a portion of the VAR scene (See Swaminathan: Figs. 5-6, and [0063], “Referring to FIG. 6A, with further reference to FIG. 5, an exemplary illustration of the image 502 with an AR object tag 504-1 displayed as a function of the location of the user's gaze is shown. FIG. 6A continues the example of FIG. 5 with the incorporation of the user's eye gaze information. The display 408 includes the image 502, a single object tag 504-1, an image segment including the Empire State Building 505, and an area of interest 506 determined by processing the user's eye gaze. In contrast to the several object tags 504 shown in FIG. 5, the single object tag 504-1 is shown based on its proximity to the area of interest 506. The device 402 may be configured to utilize the back-facing camera 406 and an eye gaze tracking algorithm (i.e., stored in memory 114) to identify the area of interest 506 based on the location of the user's gaze. The several object tags 504 may remain hidden until the area of interest 506 passes over or near a position that is associated with an AR target object. In this example, as depicted in FIG. 6A, the "Empire State Building" text is an object tag and can appear as the user's gaze passes over or near a segment of the image containing the Empire State Building. The image segment may be highlighted with a boundary line or other graphical enhancement (e.g., brightened, color change, raised) to indicate that augmented reality information is available. In the FIG. 6A, however, such highlighting is not illustrated, nor is a segment line surrounding the Empire State Building. In an embodiment, the location of the object tag maintains a position that is on or near the associated image segment such that the object tag will move if the image moves (e.g., when the orientation of the camera changes). The distance between the area of interest 506 and the object tag 504-1 may be based on a Cartesian coordinate system (e.g., pixels on the display 408). An image segmentation and recognition process may be used to recognize an AR target object and then make the association to one or more object tags. In an embodiment, the object tag 504-1 may indicate that additional augmented reality information is available. In the example of FIG. 6A, if the user prolongs their gaze (i.e., lingers) on the segment of the image containing the Empire State Building, or the "Empire State Building" text (i.e., the object tag), additional augmented reality information may be presented to the user. As previously described, the augmented reality information can be, but is not limited to, additional information which appears on the screen, such as media files, hyperlinks, GUI objects, interactive icons, executing one or more applications, or other Augmented Reality feature as known in the art”. Note that the area of interest and the gaze direction from the eye location to the scene location and checking the gaze lingering and overlap, is mapped to “determining the viewer projection vector corresponds with the location within the VAR scene based at least in part on the position of the user's eyes with respect to the VAR scene”. Note that the additional information such as medial files, is mapped to the dynamic content).
Regarding claim 5, Swaminathan and Jiang teach all the features with respect to claim 1 as outlined above. Further, Swaminathan teaches that the computer-implemented method of claim 1, wherein determining a location within the VAR scene at which to provide dynamic content comprises determining the location with respect to an object in the VAR scene (See Swaminathan: Figs. 5-6, and [0063], “Referring to FIG. 6A, with further reference to FIG. 5, an exemplary illustration of the image 502 with an AR object tag 504-1 displayed as a function of the location of the user's gaze is shown. FIG. 6A continues the example of FIG. 5 with the incorporation of the user's eye gaze information. The display 408 includes the image 502, a single object tag 504-1, an image segment including the Empire State Building 505, and an area of interest 506 determined by processing the user's eye gaze. In contrast to the several object tags 504 shown in FIG. 5, the single object tag 504-1 is shown based on its proximity to the area of interest 506. The device 402 may be configured to utilize the back-facing camera 406 and an eye gaze tracking algorithm (i.e., stored in memory 114) to identify the area of interest 506 based on the location of the user's gaze. The several object tags 504 may remain hidden until the area of interest 506 passes over or near a position that is associated with an AR target object. In this example, as depicted in FIG. 6A, the "Empire State Building" text is an object tag and can appear as the user's gaze passes over or near a segment of the image containing the Empire State Building. The image segment may be highlighted with a boundary line or other graphical enhancement (e.g., brightened, color change, raised) to indicate that augmented reality information is available. In the FIG. 6A, however, such highlighting is not illustrated, nor is a segment line surrounding the Empire State Building. In an embodiment, the location of the object tag maintains a position that is on or near the associated image segment such that the object tag will move if the image moves (e.g., when the orientation of the camera changes). The distance between the area of interest 506 and the object tag 504-1 may be based on a Cartesian coordinate system (e.g., pixels on the display 408). An image segmentation and recognition process may be used to recognize an AR target object and then make the association to one or more object tags. In an embodiment, the object tag 504-1 may indicate that additional augmented reality information is available. In the example of FIG. 6A, if the user prolongs their gaze (i.e., lingers) on the segment of the image containing the Empire State Building, or the "Empire State Building" text (i.e., the object tag), additional augmented reality information may be presented to the user. As previously described, the augmented reality information can be, but is not limited to, additional information which appears on the screen, such as media files, hyperlinks, GUI objects, interactive icons, executing one or more applications, or other Augmented Reality feature as known in the art”. Note that the area of interest and the gaze direction from the eye location to the scene location and checking the gaze lingering and overlap, is mapped to “determining the viewer projection vector corresponds with the location within the VAR scene based at least in part on the position of the user's eyes with respect to the VAR scene”. Note that the lingering on the location or object is mapped to determining the location with respect to an object in the VAR scene).
Regarding claim 6, Swaminathan and Jiang teach all the features with respect to claim 1 as outlined above. Further, Swaminathan teaches that the computer-implemented method of claim 1, wherein detecting that a viewer projection vector corresponds with the location within the VAR scene comprises processing data from the one or more sensors, the one or more sensors comprising at least one of an inertial measurement unit, a magnetometer, an accelerometer, or a camera (See Jiang: Fig. 3, and [0047], “Where the device 300 is a mobile platform, the device 300 further includes a means for capturing an image of a planar object, such as camera 114 and may optionally include motion sensors 111, such as accelerometers, gyroscopes, electronic compass, or other similar motion sensing elements. The device 300 may further includes a user interface 150 that includes a means for displaying the image and AR objects, such as the display 112. The user interface 150 may also include a keypad 152 or other input device through which the user can input information into the device 300. If desired, the keypad 152 may be obviated by integrating a virtual keypad into the display 112 with a touch sensor. The user interface 150 may also include a microphone 154 and speaker 156, e.g., if the device 300 is a mobile platform such as a cellular telephone. Of course, device 300 may include other elements unrelated to the present disclosure, such as a satellite position system receiver”).
Regarding claim 8, Swaminathan and Jiang teach all the features with respect to claim 1 as outlined above. Further, Swaminathan teaches that the computer-implemented method of claim 1, wherein the dynamic content comprises a video or an image (See Swaminathan: Figs. 5-6, and [0063], “Referring to FIG. 6A, with further reference to FIG. 5, an exemplary illustration of the image 502 with an AR object tag 504-1 displayed as a function of the location of the user's gaze is shown. FIG. 6A continues the example of FIG. 5 with the incorporation of the user's eye gaze information. The display 408 includes the image 502, a single object tag 504-1, an image segment including the Empire State Building 505, and an area of interest 506 determined by processing the user's eye gaze. In contrast to the several object tags 504 shown in FIG. 5, the single object tag 504-1 is shown based on its proximity to the area of interest 506. The device 402 may be configured to utilize the back-facing camera 406 and an eye gaze tracking algorithm (i.e., stored in memory 114) to identify the area of interest 506 based on the location of the user's gaze. The several object tags 504 may remain hidden until the area of interest 506 passes over or near a position that is associated with an AR target object. In this example, as depicted in FIG. 6A, the "Empire State Building" text is an object tag and can appear as the user's gaze passes over or near a segment of the image containing the Empire State Building. The image segment may be highlighted with a boundary line or other graphical enhancement (e.g., brightened, color change, raised) to indicate that augmented reality information is available. In the FIG. 6A, however, such highlighting is not illustrated, nor is a segment line surrounding the Empire State Building. In an embodiment, the location of the object tag maintains a position that is on or near the associated image segment such that the object tag will move if the image moves (e.g., when the orientation of the camera changes). The distance between the area of interest 506 and the object tag 504-1 may be based on a Cartesian coordinate system (e.g., pixels on the display 408). An image segmentation and recognition process may be used to recognize an AR target object and then make the association to one or more object tags. In an embodiment, the object tag 504-1 may indicate that additional augmented reality information is available. In the example of FIG. 6A, if the user prolongs their gaze (i.e., lingers) on the segment of the image containing the Empire State Building, or the "Empire State Building" text (i.e., the object tag), additional augmented reality information may be presented to the user. As previously described, the augmented reality information can be, but is not limited to, additional information which appears on the screen, such as media files, hyperlinks, GUI objects, interactive icons, executing one or more applications, or other Augmented Reality feature as known in the art”. Note that he media files are mapped to the dynamic video).
Regarding claim 9, Swaminathan and Jiang teach all the features with respect to claim 1 as outlined above. Further, Swaminathan and Jiang teach that a system (See Swaminathan: Fig. 1, and [0041], “FIG. 1 illustrates an exemplary computer system incorporating parts of the device employed in practicing embodiments of the invention. A computer system as illustrated in FIG. 1 may be incorporated as part of the computerized devices described below. For example, computer system 100 can represent some of the components of a mobile device or a smart phone. A mobile device may be any computing device with an input sensory unit like a camera and may also include a display unit. Examples of a mobile device include but are not limited to video game consoles, tablets, smart phones and any other hand-held devices. FIG. 1 provides a schematic illustration of one embodiment of a computer system 100 that can perform the methods provided by various other embodiments, as described herein, and/or can function as the host computer system, a remote kiosk/terminal, a point-of-sale device, a mobile device, a set-top box and/or a computer system. FIG. 1 is meant only to provide a generalized illustration of various components, any or all of which may be utilized as appropriate. FIG. 1, therefore, broadly illustrates how individual system elements may be implemented in a relatively separated or relatively more integrated manner”) comprising:
at least one processor (See Swaminathan: Fig. 1, and [0042], “The computer system 100 is shown comprising hardware elements that can be electrically coupled via a bus 102 (or may otherwise be in communication, as appropriate). The hardware elements may include one or more processors 104, including without limitation one or more general-purpose processors and/or one or more special-purpose processors (such as digital signal processing chips, graphics acceleration processors, and/or the like); one or more input devices 108, which can include without limitation one or more cameras, sensors (including inertial sensors), a mouse, a keyboard and/or the like; and one or more output devices 110, which can include without limitation a display unit such as the device display of FIGS. 4-10. In addition to the input device(s) 108, the front-facing camera 204 and the back-facing camera 206 as depicted in FIG. 2, may be implemented as a first camera 120 and a second camera 122, respectively. Furthermore, in some embodiments an input and output device may be integrated, for example in a touch screen or capacitive display”); and
at least one non-transitory computer readable storage medium storing instructions that, when executed by the at least one processor, cause the system (See Swaminathan: Fig. 1, and [0043], “The computer system 100 may further include (and/or be in communication with) one or more non-transitory storage devices 106, which can comprise, without limitation, local and/or network accessible storage, and/or can include, without limitation, a disk drive, a drive array, an optical storage device, a solid-state storage device such as a random access memory ("RAM") and/or a read-only memory ("ROM"), which can be programmable, flash-updateable and/or the like. Such storage devices may be configured to implement any appropriate data storage, including without limitation, various file systems, database structures, and/or the like. Device storage may be used in a number of embodiments discussed herein”) to:
determine a location within a virtual or augmented reality (VAR) scene at which to provide dynamic content (See Swaminathan: Figs. 8-13, and [0089], “At stage 1302 the device 402 may obtain an image with the front-facing camera. In an embodiment, image 702 may be a street view that is presented on the display 408 in a camera view or AR view mode. As depicted in FIG. 8, the target image may be a file displayed in a browser or other application (e.g., Google Streetview, Yelp Monocle). The device 402 may exchange location information with a location based services system. For example, the location information can be based on a GPS position, WiFi based position, or other positioning techniques as known in the art. The location information may also include an approximate heading as derived from onboard GPS chips and other internal sensors. A location based services system can provide augmented reality information to the device 402 based on the location information. For example, referring to FIG. 7, the location based services system can provide object tags 704 which are associated with locations depicted in the image 702”);
provide the VAR scene for display on a viewer device (See Swaminathan: Figs. 1-4, and [0059], “Referring to FIG. 4, with further reference to FIGS. 1 and 2, an exemplary illustration of using eye gaze tracking to identify an area of interest on an image captured with a front-facing camera on a device 402 is shown. In an embodiment, the device 402 is a mobile device 100. The device 402 includes a front-facing camera (not shown), a back-facing camera 406, and a display 408. The image in the display 408 includes a road sign 404, a background light enhancing region 410, a graphical representation of an area of interest 412, and an area of interest 414. In operation, a user can direct the front-facing camera toward a real time scene such as a road sign 404. The image can be presented on the display 408 in a camera view or an AR view mode. The image can also be stored in memory 106. The back-facing camera 406 may be used to determine an area of interest 414 based on an eye gaze tracking algorithm (e.g., FIG. 3). In an embodiment, the display 408 can activate a background light enhancing region 410 to help illuminate the user's eyes. For example, a region of the display 408 can be set to a bright white color to provide more light towards the user's face and eyes. In an embodiment, the region 410 can form a frame around the image. Other shapes and patterns such as circles or bright dots, or different shapes in the corners of the display 408 may be used. The shape of the enhancing region 410 can be used to help the camera 406 detect a key feature at stage 310 in an eye gazing process 300”);
detect (See Swaminathan: Figs. 3-4, and [0058], “Referring to FIG. 3, an exemplary prior art process flow 300 for performing eye gaze tracing with a back-facing camera on a mobile device is shown. The process 300, however, is exemplary only and not limiting. The process 300 may be altered, e.g., by having stages added, removed, or rearranged. Other process for receiving eye gaze tracking information may be used. At stage 302, the mobile device 200 can utilize the back-facing camera 206 to provide a video frame to one or more processors 104. The process can include an optional step 304 of utilizing a face detection algorithm to identify the face of the user to pass the coordinates of the eyes to an eye detection algorithm 306. The eye gaze tracking information can be based on the relative position of a user's iris. For example, the eye detection algorithm can pass the coordinates of the eyes to an iris detection algorithm at stage 308. The coordinates of the iris can be passed to a pre-processing algorithm to extract key features from the eye at stage 310. For example, the size and point of the iris can be mapped and a segment of fixed size can be selected. The brightness of each pixel within the segment can be used and input value to a learning algorithm at stage 312. By means of an example, a neural network may be used for the learning algorithm. In an example, the neural network is a basic two-layer network with a symmetric sigmoid activation function. Additional layers can be used to increase the accuracy of the neural network. Two output neurons can be used for the (x,y) screen coordinates at stage 314. The screen coordinates can be the basis for an area of interest which can be used by an application running on the device 100”; and [0059], “Referring to FIG. 4, with further reference to FIGS. 1 and 2, an exemplary illustration of using eye gaze tracking to identify an area of interest on an image captured with a front-facing camera on a device 402 is shown. In an embodiment, the device 402 is a mobile device 100. The device 402 includes a front-facing camera (not shown), a back-facing camera 406, and a display 408. The image in the display 408 includes a road sign 404, a background light enhancing region 410, a graphical representation of an area of interest 412, and an area of interest 414. In operation, a user can direct the front-facing camera toward a real time scene such as a road sign 404. The image can be presented on the display 408 in a camera view or an AR view mode. The image can also be stored in memory 106. The back-facing camera 406 may be used to determine an area of interest 414 based on an eye gaze tracking algorithm (e.g., FIG. 3). In an embodiment, the display 408 can activate a background light enhancing region 410 to help illuminate the user's eyes. For example, a region of the display 408 can be set to a bright white color to provide more light towards the user's face and eyes. In an embodiment, the region 410 can form a frame around the image. Other shapes and patterns such as circles or bright dots, or different shapes in the corners of the display 408 may be used. The shape of the enhancing region 410 can be used to help the camera 406 detect a key feature at stage 310 in an eye gazing process 300”. Note that using back-facing/eye-tracking camera sensors to determine gaze and region of interest is mapped to the projection vector corresponding to the scene location) that a viewer projection vector corresponds with the location within the VAR scene (See Jiang: Figs. 1-5, and [0022], “FIG. 5 illustrates a 3D point Q on a plane .pi. and the 2D projection q, q' of the 3D point Q on two respective images I, I' with different views of the plane .pi. to illustrate determining the pose of the master device 110A using the following notation”; and [0035], “For a 3D point Q on the plane .pi., its 2D projection q and the center of the camera 114, illustrated in FIG. 5 as point O, forms a ray. After intersecting the ray with the plane .pi., the coordinate for the 3D point Q can be determined. The plurality of reference points, correspond to a plurality of 3D points, which form a 3D plane. Generally, there are two solutions for homography decomposition. To select the correct solution, the two resulting 3D planes are stored and used to estimate the master device 110A pose in subsequent frames. When the average projection error for one plane is greater than the average projection error for the other plane, e.g., one is 1.2 times greater; the plane having the larger projection error is eliminated. If the initial image is roughly a front view of the planar object 102, the plane normal n may be used to select the correct 3D plane. After the correct 3D plane is selected, a world coordinate frame is defined to align the z-axis with the plane normal n with the origin on the plane. Thus, with the 3D-2D homography H determined, the pose of the master device 110A with respect to the object 102 is determined based on the rotation matrix R and translation vector t as discussed above with reference to equation 6. If desired, other pose determination techniques may be used”); and
provide, for display, the dynamic content within the VAR scene (See Swaminathan: Fig. 1, and [0003], “Mobile computing devices (mobile devices) provide users with access to a variety of information via wireless communication systems. As an example, mobile devices enabled for use with wireless cellular networks and/or wireless local area networks such as Wi-Fi or WiMAX provide users with access to vast information resources of the Internet. Mobile devices may also enable users to explore augmented reality environments which provide a real-time view of a target object merged with, or augmented by, computer generated graphical content. For example, cameras residing on-board a mobile device may be used in conjunction with a graphical display to present a user with supplemental information relating to targets of interest that are captured in a camera view of the mobile device. Such supplemental information may form an information layer that overlays real-world objects that are captured in a camera view of the mobile device. This functionality is generally referred to as Augmented Reality (AR) view mode. In an AR view, objects captured in the camera view can be transitioned to align with the display screen to allow a user easier access to the information layer”; and [0060], “In an embodiment, the area of interest 414 can be represented on the display 408 by a graphical representation such as one or more circles 412 to provide feedback to the user. The diameter of the circles 412 can be a function of the amount of time the user's gaze lingers in an area (e.g., the more time, the larger the diameter). Multiple circles can be used to create a tracking effect as the user's gaze moves across the display 408. The circles are exemplary only and not a limitation as other shapes or indications may be used”. Note that the overlay and the circle 412 may be mapped to the dynamic content) based on the viewer projection vector corresponding with the location (See Jiang: Fig. 1, and [0002], “An augmented reality system can insert virtual objects in a user's view of the real world. One key requirement of a successful augmented reality system is a tracking system that can estimate the user's position and orientation (pose) accurately relative to a reference. Otherwise, the virtual objects will appear at the wrong location or float around the environment. In a multi-user augmented reality system, the virtual objects need to appear at the same location in the environment from each user's unique perspective. Thus, each user's unique pose with respect to the environment needs to be estimated relative to the same reference”; [0019], “FIG. 2 is a flow chart describing the process of performing AR with multi-users without a previously acquired common reference. The master device 110A captures two or more images of an object 102 with the back facing camera 114 (202). It should be understood that as used herein, a captured image may be a still image or a video frame. The two or more images of the object 102 are captured by the master device 110A at different viewpoints, i.e., poses with respect to the object 102, or by different mobile platforms 110A and 110B having different viewpoints. Using the plurality of captured images, the pose (position and orientation) of the master device 110A with respect to the object 102 is determined (204). An image of the object 102, which may be one of the initial images captured by the master device 110A or a new or different image of the object 102, is then warped based on the orientation of the master device 110A to produce a reference image 104 as a front view of the object 102 (206) as illustrated by arrow 106 in FIG. 3. The reference image 104 is used as the common reference image from which each mobile platform 110 may determine its unique pose with respect to the object for tracking to perform multi-user AR (208)”; and [0042], “If desired, the original reference image 104 may be extended and the extended reference image may be distributed to the other mobile platforms. During the initialization process, the cameras of participating mobile platforms are pointed to capture images from the same part of the object 102. The master device 110A captures images from different perspectives and using the images from different perspectives, the reference image is generated, which can then be transmitted to the other users. Each mobile platform continues to capture images of the object 102 and uses the reference image to estimate the pose for a current image. If a significant part of the current image is not visible in the reference image, the reference image and current image may be merged to generate a new reference image, which can be transmitted to the other mobile platforms”. Note that inserting the virtual objects into the user’s view at the determined locations in the user’s view of the real/augmented environment, and using the reference images and poses of the master device to ensure stable dynamic content display, is mapped to the “(providing, for display, the dynamic content within the VAR scene) based on the viewer projection vector corresponding with the location”).
Regarding claim 10, Swaminathan and Jiang teach all the features with respect to claim 9 as outlined above. Further, Jiang teaches that the system of claim 9, wherein detecting that the viewer projection vector corresponds with the location within the VAR scene comprises determining the viewer projection vector intersects the location within the VAR scene (See Jiang: Figs. 1-5, and [0035], “For a 3D point Q on the plane .pi., its 2D projection q and the center of the camera 114, illustrated in FIG. 5 as point O, forms a ray. After intersecting the ray with the plane .pi., the coordinate for the 3D point Q can be determined. The plurality of reference points, correspond to a plurality of 3D points, which form a 3D plane. Generally, there are two solutions for homography decomposition. To select the correct solution, the two resulting 3D planes are stored and used to estimate the master device 110A pose in subsequent frames. When the average projection error for one plane is greater than the average projection error for the other plane, e.g., one is 1.2 times greater; the plane having the larger projection error is eliminated. If the initial image is roughly a front view of the planar object 102, the plane normal n may be used to select the correct 3D plane. After the correct 3D plane is selected, a world coordinate frame is defined to align the z-axis with the plane normal n with the origin on the plane. Thus, with the 3D-2D homography H determined, the pose of the master device 110A with respect to the object 102 is determined based on the rotation matrix R and translation vector t as discussed above with reference to equation 6. If desired, other pose determination techniques may be used”).
Regarding claim 11, Swaminathan and Jiang teach all the features with respect to claim 9 as outlined above. Further, Swaminathan teaches that the system of claim 9, wherein detecting that the viewer projection vector corresponds with the location within the VAR scene comprises:
determining a position of a user's eyes with respect to the VAR scene (See Swaminathan: Figs. 1-4, and [0040], “A presentation region for a media content item may be deployed to an augmented reality environment by a user for the purpose of defining a location (e.g., a position and/or orientation) where the media content item is presented within that augmented reality environment. An activation region for a media content item may be deployed within an augmented reality environment by a user for the purpose of defining conditions for how and/or when the presentation of the media content item at an associated presentation region may be enabled and/or initiated responsive to user eye gaze location and/or estimated mobile device position and/or orientation. An activation region can be within a user's eye gaze to initiate a transition from the augmented reality environment to a non-augmented reality view (i.e., an application). The transition can include stages based on the time the user's eye gaze lingers on the activation region”; and [0058], “Referring to FIG. 3, an exemplary prior art process flow 300 for performing eye gaze tracing with a back-facing camera on a mobile device is shown. The process 300, however, is exemplary only and not limiting. The process 300 may be altered, e.g., by having stages added, removed, or rearranged. Other process for receiving eye gaze tracking information may be used. At stage 302, the mobile device 200 can utilize the back-facing camera 206 to provide a video frame to one or more processors 104. The process can include an optional step 304 of utilizing a face detection algorithm to identify the face of the user to pass the coordinates of the eyes to an eye detection algorithm 306. The eye gaze tracking information can be based on the relative position of a user's iris. For example, the eye detection algorithm can pass the coordinates of the eyes to an iris detection algorithm at stage 308. The coordinates of the iris can be passed to a pre-processing algorithm to extract key features from the eye at stage 310. For example, the size and point of the iris can be mapped and a segment of fixed size can be selected. The brightness of each pixel within the segment can be used and input value to a learning algorithm at stage 312. By means of an example, a neural network may be used for the learning algorithm. In an example, the neural network is a basic two-layer network with a symmetric sigmoid activation function. Additional layers can be used to increase the accuracy of the neural network. Two output neurons can be used for the (x,y) screen coordinates at stage 314. The screen coordinates can be the basis for an area of interest which can be used by an application running on the device 100”);
determining an orientation of the viewer device (See Swaminathan: Figs. 1-4, and [0040], “A presentation region for a media content item may be deployed to an augmented reality environment by a user for the purpose of defining a location (e.g., a position and/or orientation) where the media content item is presented within that augmented reality environment. An activation region for a media content item may be deployed within an augmented reality environment by a user for the purpose of defining conditions for how and/or when the presentation of the media content item at an associated presentation region may be enabled and/or initiated responsive to user eye gaze location and/or estimated mobile device position and/or orientation. An activation region can be within a user's eye gaze to initiate a transition from the augmented reality environment to a non-augmented reality view (i.e., an application). The transition can include stages based on the time the user's eye gaze lingers on the activation region”; and [0058], “Referring to FIG. 3, an exemplary prior art process flow 300 for performing eye gaze tracing with a back-facing camera on a mobile device is shown. The process 300, however, is exemplary only and not limiting. The process 300 may be altered, e.g., by having stages added, removed, or rearranged. Other process for receiving eye gaze tracking information may be used. At stage 302, the mobile device 200 can utilize the back-facing camera 206 to provide a video frame to one or more processors 104. The process can include an optional step 304 of utilizing a face detection algorithm to identify the face of the user to pass the coordinates of the eyes to an eye detection algorithm 306. The eye gaze tracking information can be based on the relative position of a user's iris. For example, the eye detection algorithm can pass the coordinates of the eyes to an iris detection algorithm at stage 308. The coordinates of the iris can be passed to a pre-processing algorithm to extract key features from the eye at stage 310. For example, the size and point of the iris can be mapped and a segment of fixed size can be selected. The brightness of each pixel within the segment can be used and input value to a learning algorithm at stage 312. By means of an example, a neural network may be used for the learning algorithm. In an example, the neural network is a basic two-layer network with a symmetric sigmoid activation function. Additional layers can be used to increase the accuracy of the neural network. Two output neurons can be used for the (x,y) screen coordinates at stage 314. The screen coordinates can be the basis for an area of interest which can be used by an application running on the device 100”); and
determining the viewer projection vector corresponds with the location within the VAR scene based on the position of the user's eyes with respect to the VAR scene and the orientation of the viewer device (See Swaminathan: Figs. 1-6, and [0011], “An example of a method according to the disclosure includes displaying on a mobile device a first image captured by a first camera of the mobile device, determining a gaze of a user of the mobile device based on a second image captured by a second camera of the mobile device, determining an area of interest within the first image based on the determined gaze, and performing, at the mobile device, one or more actions associated with an augmented reality function based at least in part on the determined area of interest”; [0060], “In an embodiment, the area of interest 414 can be represented on the display 408 by a graphical representation such as one or more circles 412 to provide feedback to the user. The diameter of the circles 412 can be a function of the amount of time the user's gaze lingers in an area (e.g., the more time, the larger the diameter). Multiple circles can be used to create a tracking effect as the user's gaze moves across the display 408. The circles are exemplary only and not a limitation as other shapes or indications may be used”; and [0063], “Referring to FIG. 6A, with further reference to FIG. 5, an exemplary illustration of the image 502 with an AR object tag 504-1 displayed as a function of the location of the user's gaze is shown. FIG. 6A continues the example of FIG. 5 with the incorporation of the user's eye gaze information. The display 408 includes the image 502, a single object tag 504-1, an image segment including the Empire State Building 505, and an area of interest 506 determined by processing the user's eye gaze. In contrast to the several object tags 504 shown in FIG. 5, the single object tag 504-1 is shown based on its proximity to the area of interest 506. The device 402 may be configured to utilize the back-facing camera 406 and an eye gaze tracking algorithm (i.e., stored in memory 114) to identify the area of interest 506 based on the location of the user's gaze. The several object tags 504 may remain hidden until the area of interest 506 passes over or near a position that is associated with an AR target object. In this example, as depicted in FIG. 6A, the "Empire State Building" text is an object tag and can appear as the user's gaze passes over or near a segment of the image containing the Empire State Building. The image segment may be highlighted with a boundary line or other graphical enhancement (e.g., brightened, color change, raised) to indicate that augmented reality information is available. In the FIG. 6A, however, such highlighting is not illustrated, nor is a segment line surrounding the Empire State Building. In an embodiment, the location of the object tag maintains a position that is on or near the associated image segment such that the object tag will move if the image moves (e.g., when the orientation of the camera changes). The distance between the area of interest 506 and the object tag 504-1 may be based on a Cartesian coordinate system (e.g., pixels on the display 408). An image segmentation and recognition process may be used to recognize an AR target object and then make the association to one or more object tags. In an embodiment, the object tag 504-1 may indicate that additional augmented reality information is available. In the example of FIG. 6A, if the user prolongs their gaze (i.e., lingers) on the segment of the image containing the Empire State Building, or the "Empire State Building" text (i.e., the object tag), additional augmented reality information may be presented to the user. As previously described, the augmented reality information can be, but is not limited to, additional information which appears on the screen, such as media files, hyperlinks, GUI objects, interactive icons, executing one or more applications, or other Augmented Reality feature as known in the art”. Note that the area of interest and the gaze direction from the eye location to the scene location and checking the gaze lingering and overlap, is mapped to “determining the viewer projection vector corresponds with the location within the VAR scene based at least in part on the position of the user's eyes with respect to the VAR scene”).
Regarding claim 12, Swaminathan and Jiang teach all the features with respect to claim 9 as outlined above. Further, Swaminathan teaches that the system of claim 9, wherein providing the dynamic content within the VAR scene comprises providing the dynamic content over a portion of the VAR scene (See Swaminathan: Figs. 5-6, and [0063], “Referring to FIG. 6A, with further reference to FIG. 5, an exemplary illustration of the image 502 with an AR object tag 504-1 displayed as a function of the location of the user's gaze is shown. FIG. 6A continues the example of FIG. 5 with the incorporation of the user's eye gaze information. The display 408 includes the image 502, a single object tag 504-1, an image segment including the Empire State Building 505, and an area of interest 506 determined by processing the user's eye gaze. In contrast to the several object tags 504 shown in FIG. 5, the single object tag 504-1 is shown based on its proximity to the area of interest 506. The device 402 may be configured to utilize the back-facing camera 406 and an eye gaze tracking algorithm (i.e., stored in memory 114) to identify the area of interest 506 based on the location of the user's gaze. The several object tags 504 may remain hidden until the area of interest 506 passes over or near a position that is associated with an AR target object. In this example, as depicted in FIG. 6A, the "Empire State Building" text is an object tag and can appear as the user's gaze passes over or near a segment of the image containing the Empire State Building. The image segment may be highlighted with a boundary line or other graphical enhancement (e.g., brightened, color change, raised) to indicate that augmented reality information is available. In the FIG. 6A, however, such highlighting is not illustrated, nor is a segment line surrounding the Empire State Building. In an embodiment, the location of the object tag maintains a position that is on or near the associated image segment such that the object tag will move if the image moves (e.g., when the orientation of the camera changes). The distance between the area of interest 506 and the object tag 504-1 may be based on a Cartesian coordinate system (e.g., pixels on the display 408). An image segmentation and recognition process may be used to recognize an AR target object and then make the association to one or more object tags. In an embodiment, the object tag 504-1 may indicate that additional augmented reality information is available. In the example of FIG. 6A, if the user prolongs their gaze (i.e., lingers) on the segment of the image containing the Empire State Building, or the "Empire State Building" text (i.e., the object tag), additional augmented reality information may be presented to the user. As previously described, the augmented reality information can be, but is not limited to, additional information which appears on the screen, such as media files, hyperlinks, GUI objects, interactive icons, executing one or more applications, or other Augmented Reality feature as known in the art”. Note that the area of interest and the gaze direction from the eye location to the scene location and checking the gaze lingering and overlap, is mapped to “determining the viewer projection vector corresponds with the location within the VAR scene based at least in part on the position of the user's eyes with respect to the VAR scene”. Note that the additional information such as medial files, is mapped to the dynamic content).
Regarding claim 13, Swaminathan and Jiang teach all the features with respect to claim 9 as outlined above. Further, Swaminathan teaches that the system of claim 9, wherein determining a location within the VAR scene at which to provide dynamic content comprises determining the location with respect to an object in the VAR scene (See Swaminathan: Figs. 5-6, and [0063], “Referring to FIG. 6A, with further reference to FIG. 5, an exemplary illustration of the image 502 with an AR object tag 504-1 displayed as a function of the location of the user's gaze is shown. FIG. 6A continues the example of FIG. 5 with the incorporation of the user's eye gaze information. The display 408 includes the image 502, a single object tag 504-1, an image segment including the Empire State Building 505, and an area of interest 506 determined by processing the user's eye gaze. In contrast to the several object tags 504 shown in FIG. 5, the single object tag 504-1 is shown based on its proximity to the area of interest 506. The device 402 may be configured to utilize the back-facing camera 406 and an eye gaze tracking algorithm (i.e., stored in memory 114) to identify the area of interest 506 based on the location of the user's gaze. The several object tags 504 may remain hidden until the area of interest 506 passes over or near a position that is associated with an AR target object. In this example, as depicted in FIG. 6A, the "Empire State Building" text is an object tag and can appear as the user's gaze passes over or near a segment of the image containing the Empire State Building. The image segment may be highlighted with a boundary line or other graphical enhancement (e.g., brightened, color change, raised) to indicate that augmented reality information is available. In the FIG. 6A, however, such highlighting is not illustrated, nor is a segment line surrounding the Empire State Building. In an embodiment, the location of the object tag maintains a position that is on or near the associated image segment such that the object tag will move if the image moves (e.g., when the orientation of the camera changes). The distance between the area of interest 506 and the object tag 504-1 may be based on a Cartesian coordinate system (e.g., pixels on the display 408). An image segmentation and recognition process may be used to recognize an AR target object and then make the association to one or more object tags. In an embodiment, the object tag 504-1 may indicate that additional augmented reality information is available. In the example of FIG. 6A, if the user prolongs their gaze (i.e., lingers) on the segment of the image containing the Empire State Building, or the "Empire State Building" text (i.e., the object tag), additional augmented reality information may be presented to the user. As previously described, the augmented reality information can be, but is not limited to, additional information which appears on the screen, such as media files, hyperlinks, GUI objects, interactive icons, executing one or more applications, or other Augmented Reality feature as known in the art”. Note that the area of interest and the gaze direction from the eye location to the scene location and checking the gaze lingering and overlap, is mapped to “determining the viewer projection vector corresponds with the location within the VAR scene based at least in part on the position of the user's eyes with respect to the VAR scene”. Note that the lingering on the location or object is mapped to determining the location with respect to an object in the VAR scene).
Regarding claim 14, Swaminathan and Jiang teach all the features with respect to claim 9 as outlined above. Further, Swaminathan teaches that the system of claim 9, further comprising instructions that, when executed by the at least one processor, cause the system to cease to provide the dynamic content based on detecting the viewer projection vector no longer corresponds with the location within the VAR scene (See Swaminathan: Figs. 5-6, and [0063], “Referring to FIG. 6A, with further reference to FIG. 5, an exemplary illustration of the image 502 with an AR object tag 504-1 displayed as a function of the location of the user's gaze is shown. FIG. 6A continues the example of FIG. 5 with the incorporation of the user's eye gaze information. The display 408 includes the image 502, a single object tag 504-1, an image segment including the Empire State Building 505, and an area of interest 506 determined by processing the user's eye gaze. In contrast to the several object tags 504 shown in FIG. 5, the single object tag 504-1 is shown based on its proximity to the area of interest 506. The device 402 may be configured to utilize the back-facing camera 406 and an eye gaze tracking algorithm (i.e., stored in memory 114) to identify the area of interest 506 based on the location of the user's gaze. The several object tags 504 may remain hidden until the area of interest 506 passes over or near a position that is associated with an AR target object. In this example, as depicted in FIG. 6A, the "Empire State Building" text is an object tag and can appear as the user's gaze passes over or near a segment of the image containing the Empire State Building. The image segment may be highlighted with a boundary line or other graphical enhancement (e.g., brightened, color change, raised) to indicate that augmented reality information is available. In the FIG. 6A, however, such highlighting is not illustrated, nor is a segment line surrounding the Empire State Building. In an embodiment, the location of the object tag maintains a position that is on or near the associated image segment such that the object tag will move if the image moves (e.g., when the orientation of the camera changes). The distance between the area of interest 506 and the object tag 504-1 may be based on a Cartesian coordinate system (e.g., pixels on the display 408). An image segmentation and recognition process may be used to recognize an AR target object and then make the association to one or more object tags. In an embodiment, the object tag 504-1 may indicate that additional augmented reality information is available. In the example of FIG. 6A, if the user prolongs their gaze (i.e., lingers) on the segment of the image containing the Empire State Building, or the "Empire State Building" text (i.e., the object tag), additional augmented reality information may be presented to the user. As previously described, the augmented reality information can be, but is not limited to, additional information which appears on the screen, such as media files, hyperlinks, GUI objects, interactive icons, executing one or more applications, or other Augmented Reality feature as known in the art”. Note the hiding the AR target object when users are not gazing at the area of interest, is mapped to the current cited limitation of “ provide the dynamic content based on detecting the viewer projection vector no longer corresponds with the location within the VAR scene”).
Regarding claim 16, Swaminathan and Jiang teach all the features with respect to claim 1 as outlined above. Further, Swaminathan and Jiang teach that the non-transitory computer readable storage medium storing instructions thereon that, when executed by at least one processor, cause a computing device (See Swaminathan: Fig. 1, and [0041], “FIG. 1 illustrates an exemplary computer system incorporating parts of the device employed in practicing embodiments of the invention. A computer system as illustrated in FIG. 1 may be incorporated as part of the computerized devices described below. For example, computer system 100 can represent some of the components of a mobile device or a smart phone. A mobile device may be any computing device with an input sensory unit like a camera and may also include a display unit. Examples of a mobile device include but are not limited to video game consoles, tablets, smart phones and any other hand-held devices. FIG. 1 provides a schematic illustration of one embodiment of a computer system 100 that can perform the methods provided by various other embodiments, as described herein, and/or can function as the host computer system, a remote kiosk/terminal, a point-of-sale device, a mobile device, a set-top box and/or a computer system. FIG. 1 is meant only to provide a generalized illustration of various components, any or all of which may be utilized as appropriate. FIG. 1, therefore, broadly illustrates how individual system elements may be implemented in a relatively separated or relatively more integrated manner”) to:
determine a location within a virtual or augmented reality (VAR) scene at which to provide dynamic content (See Swaminathan: Figs. 8-13, and [0089], “At stage 1302 the device 402 may obtain an image with the front-facing camera. In an embodiment, image 702 may be a street view that is presented on the display 408 in a camera view or AR view mode. As depicted in FIG. 8, the target image may be a file displayed in a browser or other application (e.g., Google Streetview, Yelp Monocle). The device 402 may exchange location information with a location based services system. For example, the location information can be based on a GPS position, WiFi based position, or other positioning techniques as known in the art. The location information may also include an approximate heading as derived from onboard GPS chips and other internal sensors. A location based services system can provide augmented reality information to the device 402 based on the location information. For example, referring to FIG. 7, the location based services system can provide object tags 704 which are associated with locations depicted in the image 702”);
provide the VAR scene for display on a viewer device ; (See Swaminathan: Figs. 1-4, and [0059], “Referring to FIG. 4, with further reference to FIGS. 1 and 2, an exemplary illustration of using eye gaze tracking to identify an area of interest on an image captured with a front-facing camera on a device 402 is shown. In an embodiment, the device 402 is a mobile device 100. The device 402 includes a front-facing camera (not shown), a back-facing camera 406, and a display 408. The image in the display 408 includes a road sign 404, a background light enhancing region 410, a graphical representation of an area of interest 412, and an area of interest 414. In operation, a user can direct the front-facing camera toward a real time scene such as a road sign 404. The image can be presented on the display 408 in a camera view or an AR view mode. The image can also be stored in memory 106. The back-facing camera 406 may be used to determine an area of interest 414 based on an eye gaze tracking algorithm (e.g., FIG. 3). In an embodiment, the display 408 can activate a background light enhancing region 410 to help illuminate the user's eyes. For example, a region of the display 408 can be set to a bright white color to provide more light towards the user's face and eyes. In an embodiment, the region 410 can form a frame around the image. Other shapes and patterns such as circles or bright dots, or different shapes in the corners of the display 408 may be used. The shape of the enhancing region 410 can be used to help the camera 406 detect a key feature at stage 310 in an eye gazing process 300”);
detect, based on data received from one or more sensors of the viewer device (See Swaminathan: Figs. 3-4, and [0058], “Referring to FIG. 3, an exemplary prior art process flow 300 for performing eye gaze tracing with a back-facing camera on a mobile device is shown. The process 300, however, is exemplary only and not limiting. The process 300 may be altered, e.g., by having stages added, removed, or rearranged. Other process for receiving eye gaze tracking information may be used. At stage 302, the mobile device 200 can utilize the back-facing camera 206 to provide a video frame to one or more processors 104. The process can include an optional step 304 of utilizing a face detection algorithm to identify the face of the user to pass the coordinates of the eyes to an eye detection algorithm 306. The eye gaze tracking information can be based on the relative position of a user's iris. For example, the eye detection algorithm can pass the coordinates of the eyes to an iris detection algorithm at stage 308. The coordinates of the iris can be passed to a pre-processing algorithm to extract key features from the eye at stage 310. For example, the size and point of the iris can be mapped and a segment of fixed size can be selected. The brightness of each pixel within the segment can be used and input value to a learning algorithm at stage 312. By means of an example, a neural network may be used for the learning algorithm. In an example, the neural network is a basic two-layer network with a symmetric sigmoid activation function. Additional layers can be used to increase the accuracy of the neural network. Two output neurons can be used for the (x,y) screen coordinates at stage 314. The screen coordinates can be the basis for an area of interest which can be used by an application running on the device 100”; and [0059], “Referring to FIG. 4, with further reference to FIGS. 1 and 2, an exemplary illustration of using eye gaze tracking to identify an area of interest on an image captured with a front-facing camera on a device 402 is shown. In an embodiment, the device 402 is a mobile device 100. The device 402 includes a front-facing camera (not shown), a back-facing camera 406, and a display 408. The image in the display 408 includes a road sign 404, a background light enhancing region 410, a graphical representation of an area of interest 412, and an area of interest 414. In operation, a user can direct the front-facing camera toward a real time scene such as a road sign 404. The image can be presented on the display 408 in a camera view or an AR view mode. The image can also be stored in memory 106. The back-facing camera 406 may be used to determine an area of interest 414 based on an eye gaze tracking algorithm (e.g., FIG. 3). In an embodiment, the display 408 can activate a background light enhancing region 410 to help illuminate the user's eyes. For example, a region of the display 408 can be set to a bright white color to provide more light towards the user's face and eyes. In an embodiment, the region 410 can form a frame around the image. Other shapes and patterns such as circles or bright dots, or different shapes in the corners of the display 408 may be used. The shape of the enhancing region 410 can be used to help the camera 406 detect a key feature at stage 310 in an eye gazing process 300”. Note that using back-facing/eye-tracking camera sensors to determine gaze and region of interest is mapped to the projection vector corresponding to the scene location), that a viewer projection vector corresponds with the location within the VAR scene (See Jiang: Figs. 1-5, and [0022], “FIG. 5 illustrates a 3D point Q on a plane .pi. and the 2D projection q, q' of the 3D point Q on two respective images I, I' with different views of the plane .pi. to illustrate determining the pose of the master device 110A using the following notation”; and [0035], “For a 3D point Q on the plane .pi., its 2D projection q and the center of the camera 114, illustrated in FIG. 5 as point O, forms a ray. After intersecting the ray with the plane .pi., the coordinate for the 3D point Q can be determined. The plurality of reference points, correspond to a plurality of 3D points, which form a 3D plane. Generally, there are two solutions for homography decomposition. To select the correct solution, the two resulting 3D planes are stored and used to estimate the master device 110A pose in subsequent frames. When the average projection error for one plane is greater than the average projection error for the other plane, e.g., one is 1.2 times greater; the plane having the larger projection error is eliminated. If the initial image is roughly a front view of the planar object 102, the plane normal n may be used to select the correct 3D plane. After the correct 3D plane is selected, a world coordinate frame is defined to align the z-axis with the plane normal n with the origin on the plane. Thus, with the 3D-2D homography H determined, the pose of the master device 110A with respect to the object 102 is determined based on the rotation matrix R and translation vector t as discussed above with reference to equation 6. If desired, other pose determination techniques may be used”); and
provide, for display, the dynamic content within the VAR scene (See Swaminathan: Fig. 1, and [0003], “Mobile computing devices (mobile devices) provide users with access to a variety of information via wireless communication systems. As an example, mobile devices enabled for use with wireless cellular networks and/or wireless local area networks such as Wi-Fi or WiMAX provide users with access to vast information resources of the Internet. Mobile devices may also enable users to explore augmented reality environments which provide a real-time view of a target object merged with, or augmented by, computer generated graphical content. For example, cameras residing on-board a mobile device may be used in conjunction with a graphical display to present a user with supplemental information relating to targets of interest that are captured in a camera view of the mobile device. Such supplemental information may form an information layer that overlays real-world objects that are captured in a camera view of the mobile device. This functionality is generally referred to as Augmented Reality (AR) view mode. In an AR view, objects captured in the camera view can be transitioned to align with the display screen to allow a user easier access to the information layer”; and [0060], “In an embodiment, the area of interest 414 can be represented on the display 408 by a graphical representation such as one or more circles 412 to provide feedback to the user. The diameter of the circles 412 can be a function of the amount of time the user's gaze lingers in an area (e.g., the more time, the larger the diameter). Multiple circles can be used to create a tracking effect as the user's gaze moves across the display 408. The circles are exemplary only and not a limitation as other shapes or indications may be used”. Note that the overlay and the circle 412 may be mapped to the dynamic content) based on the viewer projection vector corresponding with the location (See Jiang: Fig. 1, and [0002], “An augmented reality system can insert virtual objects in a user's view of the real world. One key requirement of a successful augmented reality system is a tracking system that can estimate the user's position and orientation (pose) accurately relative to a reference. Otherwise, the virtual objects will appear at the wrong location or float around the environment. In a multi-user augmented reality system, the virtual objects need to appear at the same location in the environment from each user's unique perspective. Thus, each user's unique pose with respect to the environment needs to be estimated relative to the same reference”; [0019], “FIG. 2 is a flow chart describing the process of performing AR with multi-users without a previously acquired common reference. The master device 110A captures two or more images of an object 102 with the back facing camera 114 (202). It should be understood that as used herein, a captured image may be a still image or a video frame. The two or more images of the object 102 are captured by the master device 110A at different viewpoints, i.e., poses with respect to the object 102, or by different mobile platforms 110A and 110B having different viewpoints. Using the plurality of captured images, the pose (position and orientation) of the master device 110A with respect to the object 102 is determined (204). An image of the object 102, which may be one of the initial images captured by the master device 110A or a new or different image of the object 102, is then warped based on the orientation of the master device 110A to produce a reference image 104 as a front view of the object 102 (206) as illustrated by arrow 106 in FIG. 3. The reference image 104 is used as the common reference image from which each mobile platform 110 may determine its unique pose with respect to the object for tracking to perform multi-user AR (208)”; and [0042], “If desired, the original reference image 104 may be extended and the extended reference image may be distributed to the other mobile platforms. During the initialization process, the cameras of participating mobile platforms are pointed to capture images from the same part of the object 102. The master device 110A captures images from different perspectives and using the images from different perspectives, the reference image is generated, which can then be transmitted to the other users. Each mobile platform continues to capture images of the object 102 and uses the reference image to estimate the pose for a current image. If a significant part of the current image is not visible in the reference image, the reference image and current image may be merged to generate a new reference image, which can be transmitted to the other mobile platforms”. Note that inserting the virtual objects into the user’s view at the determined locations in the user’s view of the real/augmented environment, and using the reference images and poses of the master device to ensure stable dynamic content display, is mapped to the “(providing, for display, the dynamic content within the VAR scene) based on the viewer projection vector corresponding with the location”).
Regarding claim 17, Swaminathan and Jiang teach all the features with respect to claim 16 as outlined above. Further, Jiang teaches that the non-transitory computer readable storage medium of claim 16, wherein detecting that the viewer projection vector corresponds with the location within the VAR scene comprises determining the viewer projection vector intersects the location within the VAR scene (See Jiang: Figs. 1-5, and [0035], “For a 3D point Q on the plane .pi., its 2D projection q and the center of the camera 114, illustrated in FIG. 5 as point O, forms a ray. After intersecting the ray with the plane .pi., the coordinate for the 3D point Q can be determined. The plurality of reference points, correspond to a plurality of 3D points, which form a 3D plane. Generally, there are two solutions for homography decomposition. To select the correct solution, the two resulting 3D planes are stored and used to estimate the master device 110A pose in subsequent frames. When the average projection error for one plane is greater than the average projection error for the other plane, e.g., one is 1.2 times greater; the plane having the larger projection error is eliminated. If the initial image is roughly a front view of the planar object 102, the plane normal n may be used to select the correct 3D plane. After the correct 3D plane is selected, a world coordinate frame is defined to align the z-axis with the plane normal n with the origin on the plane. Thus, with the 3D-2D homography H determined, the pose of the master device 110A with respect to the object 102 is determined based on the rotation matrix R and translation vector t as discussed above with reference to equation 6. If desired, other pose determination techniques may be used”).
Regarding claim 18, Swaminathan and Jiang teach all the features with respect to claim 16 as outlined above. Further, Swaminathan teaches that the non-transitory computer readable storage medium of claim 16, wherein detecting that the viewer projection vector corresponds with the location within the VAR scene comprises:
determining a position of a user's eyes with respect to the VAR scene (See Swaminathan: Figs. 1-4, and [0040], “A presentation region for a media content item may be deployed to an augmented reality environment by a user for the purpose of defining a location (e.g., a position and/or orientation) where the media content item is presented within that augmented reality environment. An activation region for a media content item may be deployed within an augmented reality environment by a user for the purpose of defining conditions for how and/or when the presentation of the media content item at an associated presentation region may be enabled and/or initiated responsive to user eye gaze location and/or estimated mobile device position and/or orientation. An activation region can be within a user's eye gaze to initiate a transition from the augmented reality environment to a non-augmented reality view (i.e., an application). The transition can include stages based on the time the user's eye gaze lingers on the activation region”; and [0058], “Referring to FIG. 3, an exemplary prior art process flow 300 for performing eye gaze tracing with a back-facing camera on a mobile device is shown. The process 300, however, is exemplary only and not limiting. The process 300 may be altered, e.g., by having stages added, removed, or rearranged. Other process for receiving eye gaze tracking information may be used. At stage 302, the mobile device 200 can utilize the back-facing camera 206 to provide a video frame to one or more processors 104. The process can include an optional step 304 of utilizing a face detection algorithm to identify the face of the user to pass the coordinates of the eyes to an eye detection algorithm 306. The eye gaze tracking information can be based on the relative position of a user's iris. For example, the eye detection algorithm can pass the coordinates of the eyes to an iris detection algorithm at stage 308. The coordinates of the iris can be passed to a pre-processing algorithm to extract key features from the eye at stage 310. For example, the size and point of the iris can be mapped and a segment of fixed size can be selected. The brightness of each pixel within the segment can be used and input value to a learning algorithm at stage 312. By means of an example, a neural network may be used for the learning algorithm. In an example, the neural network is a basic two-layer network with a symmetric sigmoid activation function. Additional layers can be used to increase the accuracy of the neural network. Two output neurons can be used for the (x,y) screen coordinates at stage 314. The screen coordinates can be the basis for an area of interest which can be used by an application running on the device 100”);
determining an orientation of the viewer device (See Swaminathan: Figs. 1-4, and [0040], “A presentation region for a media content item may be deployed to an augmented reality environment by a user for the purpose of defining a location (e.g., a position and/or orientation) where the media content item is presented within that augmented reality environment. An activation region for a media content item may be deployed within an augmented reality environment by a user for the purpose of defining conditions for how and/or when the presentation of the media content item at an associated presentation region may be enabled and/or initiated responsive to user eye gaze location and/or estimated mobile device position and/or orientation. An activation region can be within a user's eye gaze to initiate a transition from the augmented reality environment to a non-augmented reality view (i.e., an application). The transition can include stages based on the time the user's eye gaze lingers on the activation region”; and [0058], “Referring to FIG. 3, an exemplary prior art process flow 300 for performing eye gaze tracing with a back-facing camera on a mobile device is shown. The process 300, however, is exemplary only and not limiting. The process 300 may be altered, e.g., by having stages added, removed, or rearranged. Other process for receiving eye gaze tracking information may be used. At stage 302, the mobile device 200 can utilize the back-facing camera 206 to provide a video frame to one or more processors 104. The process can include an optional step 304 of utilizing a face detection algorithm to identify the face of the user to pass the coordinates of the eyes to an eye detection algorithm 306. The eye gaze tracking information can be based on the relative position of a user's iris. For example, the eye detection algorithm can pass the coordinates of the eyes to an iris detection algorithm at stage 308. The coordinates of the iris can be passed to a pre-processing algorithm to extract key features from the eye at stage 310. For example, the size and point of the iris can be mapped and a segment of fixed size can be selected. The brightness of each pixel within the segment can be used and input value to a learning algorithm at stage 312. By means of an example, a neural network may be used for the learning algorithm. In an example, the neural network is a basic two-layer network with a symmetric sigmoid activation function. Additional layers can be used to increase the accuracy of the neural network. Two output neurons can be used for the (x,y) screen coordinates at stage 314. The screen coordinates can be the basis for an area of interest which can be used by an application running on the device 100”); and
determining the viewer projection vector corresponds with the location within the VAR scene based on the position of the user's eyes with respect to the VAR scene and the orientation of the viewer device (See Swaminathan: Figs. 1-6, and [0011], “An example of a method according to the disclosure includes displaying on a mobile device a first image captured by a first camera of the mobile device, determining a gaze of a user of the mobile device based on a second image captured by a second camera of the mobile device, determining an area of interest within the first image based on the determined gaze, and performing, at the mobile device, one or more actions associated with an augmented reality function based at least in part on the determined area of interest”; [0060], “In an embodiment, the area of interest 414 can be represented on the display 408 by a graphical representation such as one or more circles 412 to provide feedback to the user. The diameter of the circles 412 can be a function of the amount of time the user's gaze lingers in an area (e.g., the more time, the larger the diameter). Multiple circles can be used to create a tracking effect as the user's gaze moves across the display 408. The circles are exemplary only and not a limitation as other shapes or indications may be used”; and [0063], “Referring to FIG. 6A, with further reference to FIG. 5, an exemplary illustration of the image 502 with an AR object tag 504-1 displayed as a function of the location of the user's gaze is shown. FIG. 6A continues the example of FIG. 5 with the incorporation of the user's eye gaze information. The display 408 includes the image 502, a single object tag 504-1, an image segment including the Empire State Building 505, and an area of interest 506 determined by processing the user's eye gaze. In contrast to the several object tags 504 shown in FIG. 5, the single object tag 504-1 is shown based on its proximity to the area of interest 506. The device 402 may be configured to utilize the back-facing camera 406 and an eye gaze tracking algorithm (i.e., stored in memory 114) to identify the area of interest 506 based on the location of the user's gaze. The several object tags 504 may remain hidden until the area of interest 506 passes over or near a position that is associated with an AR target object. In this example, as depicted in FIG. 6A, the "Empire State Building" text is an object tag and can appear as the user's gaze passes over or near a segment of the image containing the Empire State Building. The image segment may be highlighted with a boundary line or other graphical enhancement (e.g., brightened, color change, raised) to indicate that augmented reality information is available. In the FIG. 6A, however, such highlighting is not illustrated, nor is a segment line surrounding the Empire State Building. In an embodiment, the location of the object tag maintains a position that is on or near the associated image segment such that the object tag will move if the image moves (e.g., when the orientation of the camera changes). The distance between the area of interest 506 and the object tag 504-1 may be based on a Cartesian coordinate system (e.g., pixels on the display 408). An image segmentation and recognition process may be used to recognize an AR target object and then make the association to one or more object tags. In an embodiment, the object tag 504-1 may indicate that additional augmented reality information is available. In the example of FIG. 6A, if the user prolongs their gaze (i.e., lingers) on the segment of the image containing the Empire State Building, or the "Empire State Building" text (i.e., the object tag), additional augmented reality information may be presented to the user. As previously described, the augmented reality information can be, but is not limited to, additional information which appears on the screen, such as media files, hyperlinks, GUI objects, interactive icons, executing one or more applications, or other Augmented Reality feature as known in the art”. Note that the area of interest and the gaze direction from the eye location to the scene location and checking the gaze lingering and overlap, is mapped to “determining the viewer projection vector corresponds with the location within the VAR scene based at least in part on the position of the user's eyes with respect to the VAR scene”).
Regarding claim 19, Swaminathan and Jiang teach all the features with respect to claim 16 as outlined above. Further, Swaminathan teaches that the non-transitory computer readable storage medium of claim 16, wherein providing the dynamic content within the VAR scene comprises providing the dynamic content over a portion of the VAR scene (See Swaminathan: Figs. 5-6, and [0063], “Referring to FIG. 6A, with further reference to FIG. 5, an exemplary illustration of the image 502 with an AR object tag 504-1 displayed as a function of the location of the user's gaze is shown. FIG. 6A continues the example of FIG. 5 with the incorporation of the user's eye gaze information. The display 408 includes the image 502, a single object tag 504-1, an image segment including the Empire State Building 505, and an area of interest 506 determined by processing the user's eye gaze. In contrast to the several object tags 504 shown in FIG. 5, the single object tag 504-1 is shown based on its proximity to the area of interest 506. The device 402 may be configured to utilize the back-facing camera 406 and an eye gaze tracking algorithm (i.e., stored in memory 114) to identify the area of interest 506 based on the location of the user's gaze. The several object tags 504 may remain hidden until the area of interest 506 passes over or near a position that is associated with an AR target object. In this example, as depicted in FIG. 6A, the "Empire State Building" text is an object tag and can appear as the user's gaze passes over or near a segment of the image containing the Empire State Building. The image segment may be highlighted with a boundary line or other graphical enhancement (e.g., brightened, color change, raised) to indicate that augmented reality information is available. In the FIG. 6A, however, such highlighting is not illustrated, nor is a segment line surrounding the Empire State Building. In an embodiment, the location of the object tag maintains a position that is on or near the associated image segment such that the object tag will move if the image moves (e.g., when the orientation of the camera changes). The distance between the area of interest 506 and the object tag 504-1 may be based on a Cartesian coordinate system (e.g., pixels on the display 408). An image segmentation and recognition process may be used to recognize an AR target object and then make the association to one or more object tags. In an embodiment, the object tag 504-1 may indicate that additional augmented reality information is available. In the example of FIG. 6A, if the user prolongs their gaze (i.e., lingers) on the segment of the image containing the Empire State Building, or the "Empire State Building" text (i.e., the object tag), additional augmented reality information may be presented to the user. As previously described, the augmented reality information can be, but is not limited to, additional information which appears on the screen, such as media files, hyperlinks, GUI objects, interactive icons, executing one or more applications, or other Augmented Reality feature as known in the art”. Note that the area of interest and the gaze direction from the eye location to the scene location and checking the gaze lingering and overlap, is mapped to “determining the viewer projection vector corresponds with the location within the VAR scene based at least in part on the position of the user's eyes with respect to the VAR scene”. Note that the additional information such as medial files, is mapped to the dynamic content).
Regarding claim 20, Swaminathan and Jiang teach all the features with respect to claim 16 as outlined above. Further, Swaminathan teaches that the non-transitory computer readable storage medium of claim 16, wherein determining a location within the VAR scene at which to provide dynamic content comprises determining the location with respect to an object in the VAR scene (See Swaminathan: Figs. 5-6, and [0063], “Referring to FIG. 6A, with further reference to FIG. 5, an exemplary illustration of the image 502 with an AR object tag 504-1 displayed as a function of the location of the user's gaze is shown. FIG. 6A continues the example of FIG. 5 with the incorporation of the user's eye gaze information. The display 408 includes the image 502, a single object tag 504-1, an image segment including the Empire State Building 505, and an area of interest 506 determined by processing the user's eye gaze. In contrast to the several object tags 504 shown in FIG. 5, the single object tag 504-1 is shown based on its proximity to the area of interest 506. The device 402 may be configured to utilize the back-facing camera 406 and an eye gaze tracking algorithm (i.e., stored in memory 114) to identify the area of interest 506 based on the location of the user's gaze. The several object tags 504 may remain hidden until the area of interest 506 passes over or near a position that is associated with an AR target object. In this example, as depicted in FIG. 6A, the "Empire State Building" text is an object tag and can appear as the user's gaze passes over or near a segment of the image containing the Empire State Building. The image segment may be highlighted with a boundary line or other graphical enhancement (e.g., brightened, color change, raised) to indicate that augmented reality information is available. In the FIG. 6A, however, such highlighting is not illustrated, nor is a segment line surrounding the Empire State Building. In an embodiment, the location of the object tag maintains a position that is on or near the associated image segment such that the object tag will move if the image moves (e.g., when the orientation of the camera changes). The distance between the area of interest 506 and the object tag 504-1 may be based on a Cartesian coordinate system (e.g., pixels on the display 408). An image segmentation and recognition process may be used to recognize an AR target object and then make the association to one or more object tags. In an embodiment, the object tag 504-1 may indicate that additional augmented reality information is available. In the example of FIG. 6A, if the user prolongs their gaze (i.e., lingers) on the segment of the image containing the Empire State Building, or the "Empire State Building" text (i.e., the object tag), additional augmented reality information may be presented to the user. As previously described, the augmented reality information can be, but is not limited to, additional information which appears on the screen, such as media files, hyperlinks, GUI objects, interactive icons, executing one or more applications, or other Augmented Reality feature as known in the art”. Note that the area of interest and the gaze direction from the eye location to the scene location and checking the gaze lingering and overlap, is mapped to “determining the viewer projection vector corresponds with the location within the VAR scene based at least in part on the position of the user's eyes with respect to the VAR scene”. Note that the lingering on the location or object is mapped to determining the location with respect to an object in the VAR scene)..
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Swaminathan, etc. (US 20140168056 A1) in view of Jiang, etc. (US 20120062702 A1), further in view of Crocco, etc. (US 20130162505 A1).
Regarding claim 7, Swaminathan and Jiang teach all the features with respect to claim 2 as outlined above. However, Swaminathan, modified by Jiang, fails to explicitly disclose that the computer-implemented method of claim 2, further comprising providing, for display on the viewer device, the VAR scene by providing a stereo pair of VAR scene content for display on the viewer device.
However, Crocco teaches that the computer-implemented method of claim 2, further comprising providing, for display on the viewer device, the VAR scene by providing a stereo pair of VAR scene content for display on the viewer device (See Crocco: Fig. 1, and [0051], “In another embodiment, the HMD device 102 provides passive stereoscopic vision. Since the environmental-light filters 100 may be polarized, right and left lenses 104 can be oriented so that the polarization is different by 90 degrees. As such, the HMD device 102 equipped with the environmental-light filters 104 can be used with a comparably equipped 3-D display to view images in 3-D”; and Figs. 7A-F, and [0065], “FIG. 7F depicts a combined image 710 that might be viewed by a user with a substantially opaque environmental-light filter removably coupled to an HMD device. In an embodiment, the substantially opaque environmental-light filter has a light transmissivity between about 15 and about 0%. As such, the augmented-reality image 702 appears opaque or solid and the real-world scene 700 is only visible to a very small extent or is not visible”. Note that 3D scene with depth information is provided and displayed to the user, and this is mapped to the stereo pairs of scene for the user).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was effectively filed to modify Swaminathan to have the computer-implemented method of claim 2, further comprising providing, for display on the viewer device, the VAR scene by providing a stereo pair of VAR scene content for display on the viewer device as taught by Crocco in order to enhance appearance of the augmented-reality image by reducing or eliminating environmental light that reaches the user's eye (See Crocco: Figs. 7A-F, and [0066], “As depicted in FIGS. 7A-F, environmental-light filters are selectable based on a level of light transmissivity to provide a desired appearance in a combined image. For example, a user might select a mild filter to enhance an augmented-reality image to a small extent, such as when the augmented-reality image is only being viewed periodically or nonchalantly, e.g. the augmented-reality image may provide a heads-up display of the current time and date which the user only periodically focuses his or her attention on. Or the user might select a mild filter on a cloudy day or when in low environmental lighting conditions because greater filtration of the environmental light is not needed. Such, may provide the user with a desired viewing experience of the augmented-reality image while also not obstructing the user's ability to view the real-world scene”). Swaminathan teaches a method and system that may generate and display scene images to the user based on the region of interest gazed by the user’s eye with gaze direction detection algorithm; while Crocco teaches an environmental-light filter removably coupled to an optical see-through head-mounted display (HMD) device and method that may filter the environmental light and provide stereo effect scene to the user. Therefore, it is obvious to one of ordinary skill in the art to modify Swaminathan by Crocco to filter the environmental light and provide stereo pairs of scene images to the HMD user to enhance the realistic environment appearance. The motivation to modify Swaminathan by Crocco is “Use of known technique to improve similar devices (methods, or products) in the same way”.
Claim 15 is rejected under 35 U.S.C. 103 as being unpatentable over Swaminathan, etc. (US 20140168056 A1) in view of Jiang, etc. (US 20120062702 A1), further in view of McArdle, etc. (US 20120218306 A1).
Regarding claim 15, Swaminathan and Jiang teach all the features with respect to claim 9 as outlined above. Further, Swaminathan teaches that the system of claim 9, wherein:
the VAR scene comprises a spherical spatial image; and
the dynamic content comprises an image or a video (See Swaminathan: Figs. 5-6, and [0063], “Referring to FIG. 6A, with further reference to FIG. 5, an exemplary illustration of the image 502 with an AR object tag 504-1 displayed as a function of the location of the user's gaze is shown. FIG. 6A continues the example of FIG. 5 with the incorporation of the user's eye gaze information. The display 408 includes the image 502, a single object tag 504-1, an image segment including the Empire State Building 505, and an area of interest 506 determined by processing the user's eye gaze. In contrast to the several object tags 504 shown in FIG. 5, the single object tag 504-1 is shown based on its proximity to the area of interest 506. The device 402 may be configured to utilize the back-facing camera 406 and an eye gaze tracking algorithm (i.e., stored in memory 114) to identify the area of interest 506 based on the location of the user's gaze. The several object tags 504 may remain hidden until the area of interest 506 passes over or near a position that is associated with an AR target object. In this example, as depicted in FIG. 6A, the "Empire State Building" text is an object tag and can appear as the user's gaze passes over or near a segment of the image containing the Empire State Building. The image segment may be highlighted with a boundary line or other graphical enhancement (e.g., brightened, color change, raised) to indicate that augmented reality information is available. In the FIG. 6A, however, such highlighting is not illustrated, nor is a segment line surrounding the Empire State Building. In an embodiment, the location of the object tag maintains a position that is on or near the associated image segment such that the object tag will move if the image moves (e.g., when the orientation of the camera changes). The distance between the area of interest 506 and the object tag 504-1 may be based on a Cartesian coordinate system (e.g., pixels on the display 408). An image segmentation and recognition process may be used to recognize an AR target object and then make the association to one or more object tags. In an embodiment, the object tag 504-1 may indicate that additional augmented reality information is available. In the example of FIG. 6A, if the user prolongs their gaze (i.e., lingers) on the segment of the image containing the Empire State Building, or the "Empire State Building" text (i.e., the object tag), additional augmented reality information may be presented to the user. As previously described, the augmented reality information can be, but is not limited to, additional information which appears on the screen, such as media files, hyperlinks, GUI objects, interactive icons, executing one or more applications, or other Augmented Reality feature as known in the art”. Note that he media files are mapped to the dynamic video).
However, Swaminathan, modified by Jiang, fails to explicitly disclose that the system of claim 9, wherein: the VAR scene comprises a spherical spatial image.
However, McArdle teaches that the system of claim 9, wherein: the VAR scene comprises a spherical spatial image (See McArdle: Fig. 5, and [0040], “As shown in FIG. 5, in another variation of the device 14 of the preferred embodiment, the scene can include a spherical image 20. Preferably, the portion of the spherical image (i.e., the scene 18) that is displayable by the device 14 corresponds to an overlap between a viewing frustum of the device (i.e., a viewing cone projected from the device) and the imaginary sphere that includes the spherical image 20. The scene 18 is preferably a portion of the spherical image 20, which can include a substantially rectangular display of a concave, convex, or hyperbolic rectangular portion of the sphere of the spherical image 20. Preferably, the nodal point is disposed at approximately the origin of the spherical image 20, such that a user 12 has the illusion of being located at the center of a larger sphere or bubble having the VAR scene displayed on its interior. Alternatively, the nodal point can be disposed at any other suitable vantage point within the spherical image 20 displayable by the device 14. In another alternative, the displayable scene can include a substantially planar and/or ribbon-like geometry from which the nodal point is distanced in a constant or variable fashion. Preferably, the display of the scene 18 can be performed within a 3D or 2D graphics platform such as OpenGL, WebGL, or Direct 3D. Alternatively, the display of the scene 18 can be performed within a browser environment using one or more of HTML5, CSS3, or any other suitable markup language. In another variation of the device 14 of the preferred embodiment, the geometry of the displayable scene can be altered and/or varied in response to an automated input and/or in response to a user input”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was effectively filed to modify Swaminathan to have the system of claim 9, wherein: the VAR scene comprises a spherical spatial image as taught by McArdle in order to enable orienting the scene to prepare a VAR scene for display such that the viewable scene matches the user view in a real 3D view and the displayable scene provides a simulation of real viewable space to the user if the device is provided with a transparent frame, thus displaying reality elements in addition to other real elements e.g. buildings, roads and landmarks, in an enhanced manner (See McArdle: Fig. 1, and [0020], “As shown in FIG. 1, the method of the preferred embodiment can also include block S104, which recites orienting a scene displayable on the device to a user in response to the real orientation and the user orientation. Block S104 preferably functions to process, compute, calculate, determine, and/or create a VAR scene that can be displayed on the device to a user, wherein the VAR scene is oriented to mimic the effect of the user viewing the VAR scene as if through the frame of the device. Preferably, orienting the scene can include preparing a VAR scene for display such that the viewable scene matches what the user would view in a real three-dimensional view, that is, such that the displayable scene provides a simulation of real viewable space to the user as if the device were a transparent frame. As noted above, the scene is preferably a VAR scene, therefore it can include one or more virtual and/or augmented reality elements composing, in addition to, and/or in lieu of one or more real elements (buildings, roads, landmarks, and the like, either real or fictitious). Alternatively, the scene can include processed or unprocessed images/videos/multimedia files of a multitude of scene aspects, including both actual and fictitious elements as noted above”). Swaminathan teaches a method and system that may generate and display scene images to the user based on the region of interest gazed by the user’s eye with gaze direction detection algorithm; while McArdle teaches a system and method that may present a scene to a user according to a preferred embodiment includes determining a real orientation of a device relative to a projection matrix and determining a user orientation of the device relative to a nodal point using the spherical coordinate system in order to have more realistic view of the environment for the user. Therefore, it is obvious to one of ordinary skill in the art to modify Swaminathan by McArdle present the environment scene images in the spherical spatial image. The motivation to modify Swaminathan by McArdle is “Use of known technique to improve similar devices (methods, or products) in the same way”.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to GORDON G LIU whose telephone number is (571)270-0382. The examiner can normally be reached Monday - Friday 8:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Devona E Faulk can be reached at 571-272-7515. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/GORDON G LIU/ Primary Examiner, Art Unit 2618
Prosecution Insights