DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Application
This office action is in response to the most recent filings filed by applicants on 01/28/26. Preliminary amendments filed on 01/28/26 are the ones considered in the Office action below. The office action below is Non-Final Action.
No claims are amended
Claims 1-13 are cancelled
Claims 14-33 are newly added
Claims 14-33 are pending
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 14-33 is/are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception (i.e., an abstract idea) without significantly more.
Step One - First, pursuant to step 1 in the January 2019 Guidance on 84 Fed. Reg. 53, the claims 14-20 is/are directed to a method which is a statutory category.
Step One - First, pursuant to step 1 in the January 2019 Guidance on 84 Fed. Reg. 53, the claims 21-27 is/are directed to a non-transitory computer readable medium which is a statutory category.
Step One - First, pursuant to step 1 in the January 2019 Guidance on 84 Fed. Reg. 53, the claims 28-33 is/are directed to a system which is a statutory category.
Step 2A Prong 1: Identify the Abstract Idea(s)
The Alice framework, steps 2A-Prong One (part 1 of Mayo Test), here, the claims are analyzed to determine if the claims are directed to a judicial exception. MPEP 2106.04(a). In determining whether the claims are directed to a judicial exception, the claims are analyzed to evaluate whether the claims recite a judicial exception (Prong One of Step 2A), and whether the claims recite additional elements that integrate the judicial exception into a practical application (Prong Two of Step 2A). See 2019 Revised Patent Subject Matter Eligibility Guidance (“PEG” 2019 Revised Patent Subject Matter Eligibility Guidance, 84 Fed. Reg. 50-57 (Jan. 7, 2019)).
Under the 2019 PEG, Step 2A under which a claim is not “directed to” a judicial exception unless the claim satisfies a two-prong inquiry. Further, particular groupings of abstract ideas are consistent with judicial precedent and are based on an extraction and synthesis of the key concepts identified by the courts as being abstract.
Independent claims 14, 21 and 28, with respect to the Step 2A, Prong One, when “taken as a whole” the claims as drafted, and given their broadest reasonable interpretation, fall within the Abstract idea grouping of “certain methods of organizing human activity” (business relations; relationships or interactions between people). For instance, independent Method Claim 14 is directed to an abstract idea, as evidenced by claim limitations “collecting eye heading information that come into proximity with an object in an environment, wherein the eye heading information is collected by capturing an image of an eye of a wearer; and generating a map of the environment with an indication apparently looked at the object, as understood by examining the eye heading information.”
In the originally submitted specification, the claimed invention is described as: [0069] Aspects of the present invention relate to head-worn computing ("HWC") systems. HWC involves, in some instances, a system that mimics the appearance of head-worn glasses or sunglasses. The glasses may be a fully developed computing platform, such as including computer displays presented in each of the lenses of the glasses to the eyes of the user. In embodiments, the lenses and displays may be configured to allow a person wearing the glasses to see the environment through the lenses while also seeing, simultaneously, digital imagery, which forms an overlaid image that is perceived by the person as a digitally augmented image of the environment, or augmented reality ("AR").
These claim limitations belong to the grouping of “certain methods of organizing human activity” because the claims are related to managing what the person is seeing the environment through the lenses while also seeing, simultaneously for one or more human entities involves organizing human activity based on the description of “certain methods of organizing human activity” provided by the courts. The court have used the phrase “Certain methods of organizing human activity” as —fundamental economic principles or practices (including hedging, insurance, mitigating risk); commercial or legal interactions (including agreements in the form of contracts; legal obligations; advertising, marketing or sales activities or behaviors; business relations); managing personal behavior or relationships or interactions between people (including social activities, teaching, and following rules or instructions).
Independent Claims 21 and 28 is/are recite substantially similar limitations to independent claim 14 and is/are rejected under 2A for similar reasons to claim 14 above.
Step 2A Prong 2: Additional Elements That Integrate the Judicial Exception into a Practical Application
With respect to the Step 2A, Prong Two - This judicial exception is not integrated into a practical application. In particular, the claim recites additional elements: “A method, comprising: A non-transitory computer-readable medium storing instructions which, when executed by one or more processors cause the one or more processors to perform a method comprising: A system comprising: one or more processors configured to: from a plurality of head-worn computers, from each of the plurality of head-worn computers, of the head-worn computers through an optical path internal to content display optics of each head-worn computer, of how many of the plurality of head-worn computers, from the plurality of head-worn computers” at a high level of generality such that it amounts to no more than: adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea, as discussed in MPEP 2106.05(f). Accordingly, these additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea. The claims are directed to an abstract idea with no significantly more elements.
Thus, the additional elements do not integrate the abstract idea into practical application because they do not impose any meaningful limitations on practicing the abstract idea. As a result, claims 14, 21 and 28 do not provide any specifics regarding the integration into a practical application when recited in a claim with a judicial exception. See MPEP 2106.05(f).
Applicants originally submitted specification describes the computer components above at least in page/ paragraph [0071], [0073]-[0079]. In light of the specification, it should be noted that the components discussed above did not meaningfully limit the abstract idea because they merely linked the use of the abstract idea to a particular technological environment (i.e., "implementation via computers").
Similarly dependent claims 15-20, 22-27, and 29-33 are also directed to an abstract idea under 2A, first and second prong. In the present application, all of the dependent claims have been evaluated, and it was found that they all inherit the deficiencies set forth with respect to the independent claims. For instance, dependent claims 17 recite “wherein the map comprises a second indication of the eye heading information”. Here, these claims offer further descriptive limitations of elements found in the independent claims which are similar to the abstract idea noted in the independent claim above.
Dependent claims 15 recites “wherein the map is generated using the respective locations and movement paths of each of the plurality of head-worn computers.” Dependent claims 16 recites “wherein the map is generated using the respective locations and movement paths of each of the plurality of head-worn computers as the each of the plurality of head-worn computers moves relative to the object”. Dependent claims 18 recites “wherein the map comprises a third indication of respective locations of each of the plurality of head-worn computers when the object was in respective field of views of each of the plurality of head-worn computers”. Dependent claims 19 recites “wherein the map comprises a fourth indication of respective lengths of time the object was in respective field of views of each of the plurality of head-worn computers”. Dependent claims 20 recites “wherein the map comprises a fifth indication of respective movements of each of the plurality of head-worn computers after the object exits respective field of views of each of the plurality of head-worn computers”. In this claim, “plurality of head-worn computers”, “ is/ are an additional element, but it is still being recited such that it amounts to no more than: adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea, as discussed in MPEP 2106.05(f). As a result, Examiner asserts that dependent claims, such as dependent claims 15-20, 22-27, and 29-33 are also directed to the abstract idea identified above.
Step 2B: Determine Whether Any Element, Or Combination, Amount to “Significantly More” Than the Abstract Idea Itself
With respect to Step 2B, the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. First, the invention lacks improvements to another technology or technical field [see Alice at 2351; 2019 IEG at 55], and lacks meaningful limitations beyond generally linking the use of an abstract idea to a particular technological environment [Alice at 2360, 2019 IEG at 55], and fails to effect a transformation or reduction of a particular article to a different state or thing [2019 IEG, 55]. For the reasons articulated above, the claims recite an abstract idea that is limited to a particular field of endeavor (MPEP § 2106.05(h)) and recites insignificant extra-solution activity (MPEP § 2106.05(g)). By the factors and rationale provided above with respect to these MPEP sections, the additional elements of the claims that fail to integrate the abstract idea into a practical application also fail to amount to “significantly more” than the abstract idea.
As discussed above with respect to integration of the abstract idea into a practical application, the additional element(s) of “A method, comprising: A non-transitory computer-readable medium storing instructions which, when executed by one or more processors cause the one or more processors to perform a method comprising: A system comprising: one or more processors configured to: from a plurality of head-worn computers, from each of the plurality of head-worn computers, of the head-worn computers through an optical path internal to content display optics of each head-worn computer, of how many of the plurality of head-worn computers, from the plurality of head-worn computers” are insufficient to amount to significantly more. Applicants originally submitted specification describes the computer components above at least in page/ paragraph [0071], [0073]-[0079]. In light of the specification, it should be noted that the components discussed above did not meaningfully limit the abstract idea because they merely linked the use of the abstract idea to a particular technological environment (i.e., "implementation via computers"). In light of the specification, it should be noted that the claim limitations discussed above are merely instructions to implement the abstract idea on a computer. See MPEP 2106.05(f). (See MPEP 2106.05(f) - Mere Instructions to Apply an Exception - “Thus, for example, claims that amount to nothing more than an instruction to apply the abstract idea using a generic computer do not render an abstract idea eligible.” Alice Corp., 134 S. Ct. at 235). Mere instructions to apply an exception using computer component cannot provide an inventive concept.). The additional elements amount to no more than a recitation of generic computer elements utilized to perform generic computer functions, such as performing repetitive calculations, Bancorp Services v. Sun Life, 687 F.3d 1266, 1278, 103 USPQ2d 1425, 1433 (Fed. Cir. 2012) ("The computer required by some of Bancorp’s claims is employed only for its most basic function, the performance of repetitive calculations, and as such does not impose meaningful limits on the scope of those claims."); and storing and retrieving information in memory, Versata Dev. Group, Inc. v. SAP Am., Inc., 793 F.3d 1306, 1334, 115 USPQ2d 1681, 1701 (Fed. Cir. 2015); OIP Techs., 788 F.3d at 1363, 115 USPQ2d at 1092-93; see MPEP 2106.05(d)(II).
Therefore, the claims at issue do not require any nonconventional computer, network, or display components, or even a “non-conventional and non-generic arrangement of know, conventional pieces,” but merely call for performance of the claimed on a set of generic computer components” and display devices. All of these additional elements are significantly more because these, again, are merely the software and/or hardware components used to implement the abstract idea on a general-purpose computer. Generically recited computer elements do not add a meaningful limitation to the abstract idea because the Alice decision noted that generic structures that merely apply abstract ideas are not significantly more than the abstract ideas.
The computing elements with a computing device is recited at high level of generality (e.g. a generic device performing a generic computer function of processing data). Thus, this step is no more than mere instructions to apply the exception on a generic computer. In addition, using a processor to process data has been well- understood routing, conventional activity in the industry for many years. Generic computer features, such as system or storage, do not amount to significantly more than the abstract idea. These limitations merely describe implementation for the invention using elements of a general-purpose system, which is not sufficient to amount to significantly more. See, e.g., Alice Corp., 134 S. Ct. 2347, 110 USPQ2d 1976; Versata Dev. Group, Inc. v. SAP Am. Inc., 793 F.3d 1306, 1334, 115 USPQ2d 1681, 1791 (Federal Circuit 2015).
Independent Claims 21 and 28 is/are recite substantially similar limitations to independent claim 14 and is/are rejected under 2B for similar reasons to claim 14 above.
Further, it should be noted that additional elements of the claimed invention such as claim limitations when considered individually or as an ordered combination along with the other limitations discussed above in method claim 14 also do not meaningfully limit the abstract idea because they merely linked the use of the abstract idea to a particular technological environment (i.e., "implementation via computers"). In light of the specification, it should be noted that the claim limitations discussed above are merely instructions to implement the abstract idea on a computer. See MPEP 2106.
Similarly, dependent claims 15-20, 22-27, and 29-33 also do not include limitations amounting to significantly more than the abstract idea under the second prong or 2B of the Alice framework. In the present application, all of the dependent claims have been evaluated and it was found that they all inherit the deficiencies set forth with respect to the independent claims. Further, it should be noted that the dependent claims do not include limitations that overcome the stated assertions. Here, the dependent claims recite features/limitations that include computer components identified above in part 2B of analysis of independent claims 14, 21 and 28. As a result, Examiner asserts that dependent claims, such as dependent claims 15-20, 22-27, and 29-33 are also directed to the abstract idea identified above.
Further, Examiner notes that the addition limitations, when considered as an ordered combination, add nothing that is not already present when looking at the additional elements individually.
For more information on 101 rejections, see MPEP 2106, January 2019 Guidance at https://www.govinfo.gov/content/pkg/FR-2019-01 -07/pdf/2018-28282.pdf
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 14-33 is/are rejected under 35 U.S.C. 103 as being unpatentable over Neven; Hartmut (US 2012/0290401), and further in view of Hennessey; Craig A. et al. (US 2014/0320397).
As per claims 14, 21 and 28: Regarding the claim limitations below, Reference Hennessey shows:
A method, comprising (Reference Hennessey shows: [0015] Embodiments of a system, apparatus, and methods for a gaze tracking system are described herein. In the following description numerous specific details are set forth to provide a thorough understanding of the embodiments. One skilled in the relevant art will recognize, however, that the techniques described herein can be practiced without one or more of the specific details, or with other methods, components, materials, etc. In other instances, well-known structures, materials, or operations are not shown or described in detail to avoid obscuring certain aspects. [0055] The processes explained above are described in terms of computer software and hardware. The techniques described may constitute machine-executable instructions embodied within a tangible machine (e.g., computer) readable storage medium, that when executed by a machine will cause the machine to perform the operations described. Additionally, the processes may be embodied within hardware, such as an application specific integrated circuit ("ASIC") or the like. [0056] A tangible machine-readable storage medium includes any mechanism that provides (i.e., stores) information in a form accessible by a machine (e.g., a computer, network device, personal digital assistant, manufacturing tool, any device with a set of one or more processors, etc.). For example, a tangible machine-readable storage medium includes one or more recordable/non-recordable media (e.g., read only memory (ROM), random access memory (RAM), magnetic disk storage media, optical storage media, flash memory devices, etc.)):
Regarding the claim limitations below, Reference Hennessey shows:
A non-transitory computer-readable medium storing instructions which, when executed by one or more processors cause the one or more processors to perform a method comprising (Reference Hennessey shows: [0015] Embodiments of a system, apparatus, and methods for a gaze tracking system are described herein. In the following description numerous specific details are set forth to provide a thorough understanding of the embodiments. One skilled in the relevant art will recognize, however, that the techniques described herein can be practiced without one or more of the specific details, or with other methods, components, materials, etc. In other instances, well-known structures, materials, or operations are not shown or described in detail to avoid obscuring certain aspects. [0055] The processes explained above are described in terms of computer software and hardware. The techniques described may constitute machine-executable instructions embodied within a tangible machine (e.g., computer) readable storage medium, that when executed by a machine will cause the machine to perform the operations described. Additionally, the processes may be embodied within hardware, such as an application specific integrated circuit ("ASIC") or the like. [0056] A tangible machine-readable storage medium includes any mechanism that provides (i.e., stores) information in a form accessible by a machine (e.g., a computer, network device, personal digital assistant, manufacturing tool, any device with a set of one or more processors, etc.). For example, a tangible machine-readable storage medium includes one or more recordable/non-recordable media (e.g., read only memory (ROM), random access memory (RAM), magnetic disk storage media, optical storage media, flash memory devices, etc.)):
Regarding the claim limitations below, Reference Hennessey shows:
A system, comprising: one or more processors (Reference Hennessey shows: [0015] Embodiments of a system, apparatus, and methods for a gaze tracking system are described herein. In the following description numerous specific details are set forth to provide a thorough understanding of the embodiments. One skilled in the relevant art will recognize, however, that the techniques described herein can be practiced without one or more of the specific details, or with other methods, components, materials, etc. In other instances, well-known structures, materials, or operations are not shown or described in detail to avoid obscuring certain aspects. [0055] The processes explained above are described in terms of computer software and hardware. The techniques described may constitute machine-executable instructions embodied within a tangible machine (e.g., computer) readable storage medium, that when executed by a machine will cause the machine to perform the operations described. Additionally, the processes may be embodied within hardware, such as an application specific integrated circuit ("ASIC") or the like. [0056] A tangible machine-readable storage medium includes any mechanism that provides (i.e., stores) information in a form accessible by a machine (e.g., a computer, network device, personal digital assistant, manufacturing tool, any device with a set of one or more processors, etc.). For example, a tangible machine-readable storage medium includes one or more recordable/non-recordable media (e.g., read only memory (ROM), random access memory (RAM), magnetic disk storage media, optical storage media, flash memory devices, etc.)):
Regarding the claim limitations below, Reference Hennessey shows:
collecting eye heading information from a plurality of head-worn computers that come into proximity with an object in an environment, wherein the eye heading information is collected from each of the plurality of head-worn computers by capturing an image of an eye of a wearer of the head-worn computers through an optical path internal to content display optics of each head-worn computer (Reference Hennessey shows: Neven shows use of a head-worn computer that can be used to gather information about the user, and Neven shows use of wearable glasses as a medium for gathering information about the user ([0021]-[0022]). Neven also shows “and an image of an eye”. Neven shows this limitation in [0065]-[0067]: where Neven shows eye tracking. Claim interpretation in light of the spec: It should be noted that this limitation is shown in applicants’ originally submitted specification at least in [0110]-[0112], [0240]. However, none of the paragraphs describe what or how the camera is disposed in line with an optical path. As such, the scope of the above claim terms is very broad and encompasses things like tracking of the eye.
In light of the specification, Neven shows the above limitation in Fig. 4, #405-415, [0003]: receiving scene images from a head mounted gaze tracking device capturing external scenes viewed by a user wearing the head mounted device. Further, in [0017], Fig. 1 illustrates eyeglasses 102 as an example of a head mounted gaze tracking device, other types of wearable computing devices could additionally or alternatively be used. Further, in [0045]: shows each identified item may even be indexed along with location information (e.g., GPS position information) if the gaze tracking device is equipped with a GPS receiver. Further, in [0021] Gazing tracking camera 121 is positioned to acquire eye images of the user's eye.
Neven also shows plurality of users in [0050]: Any of the metrics gathered by gaze tracking schemes described above may be accumulated across many users to create batched gaze tracking information. This reads on “first user” in the claim above. Neven also shows plurality of users in [0050]: Any of the metrics gathered by gaze tracking schemes described above may be accumulated across many users to create batched gaze tracking information. Fig. 5A, #515-520: [0044] In a process block 515, server system 160 executes an image recognition algorithm on the scene image to identify item(s) within the user's external scene. In one embodiment, the image recognition is executed on just the item that the user is deemed to be directly looking at as determined by the gaze direction information. In another embodiment, the image recognition algorithm is executed on multiple items in the vicinity of where the user is gazing as determined by the gaze direction information. In yet another embodiment, the image recognition algorithm is executed on all items within the captured scene image that are amenable to visual searching. [0021]: Although FIGS. 1 and 2 illustrate a single gaze tracking camera 121 positioned on the right side of eyeglasses 102, other embodiments may include multiple gaze tracking cameras (e.g., one to record video images of each eye) having other positions (e.g., positioned on any of frame elements 104, 106, 108, 114, and 116). [0050] Any of the metrics gathered by gaze tracking schemes described above may be accumulated across many users to create batched gaze tracking information. This reads on the above claim limitation.
Neven shows the above limitation in Fig. 4, #405-415, [0003]: receiving scene images from a head mounted gaze tracking device capturing external scenes viewed by a user wearing the head mounted device. Further, in [0017], Fig. 1 illustrates eyeglasses 102 as an example of a head mounted gaze tracking device, other types of wearable computing devices could additionally or alternatively be used. Further, in [0045]: shows each identified item may even be indexed along with location information (e.g., GPS position information) if the gaze tracking device is equipped with a GPS receiver. Further, in [0021] Gazing tracking camera 121 is positioned to acquire eye images of the user's eye. Also see, [0040], Fig. 3 #175: Gazing Log. Further, in [0021] Gazing tracking camera 121 is positioned to acquire eye images of the user's eye. Also see, [0040], Fig. 3 #175: Gazing Log. [0048]: For example, when a user looks directly at an identified item being tracked by the system, the user may trigger an information search (e.g., Wikipedia search, Google search, etc.) to obtain general or specific background information on the identified item being tracked by signaling with a particular eye gesture or via finger-operable touch pads 124, 126 on device 100. The information search results can then be quickly served up to the user in the heads-up display (e.g., lens elements 110 and 112). Thus, latent pre-searching can improve visual searching to identify an item by triggering the visual search when it first enters the user's peripheral view before it reaches his direct gaze and speed up potential informational searches on identified items submitted by the user. In one embodiment, once a latent pre-search has identified a particular item, the informational search may also be automatically triggered as a latent pre-search and the informational results also queued for quick, on-demand user retrieval. [0045]: In one embodiment, the indication of whether the user looked directly at an identified item is determined based upon both the gaze direction information and whether the user looked directly at the identified item for a threshold period of time to filter out glancing views. In one embodiment, each identified item may even be indexed along with location information (e.g., GPS position information) if the gaze tracking device is equipped with a GPS receiver. Of course, position information may also be acquired from local cell towers or WiFi hot spot locations at the time that the gaze tracking device transmits the scene images. Gazing log 175 may be generated and stored locally at server system 160 or archived into a database 170. [0048]: For example, when a user looks directly at an identified item being tracked by the system, the user may trigger an information search (e.g., Wikipedia search, Google search, etc.) to obtain general or specific background information on the identified item being tracked by signaling with a particular eye gesture or via finger-operable touch pads 124, 126 on device 100. The information search results can then be quickly served up to the user in the heads-up display (e.g., lens elements 110 and 112). Thus, latent pre-searching can improve visual searching to identify an item by triggering the visual search when it first enters the user's peripheral view before it reaches his direct gaze and speed up potential informational searches on identified items submitted by the user. In one embodiment, once a latent pre-search has identified a particular item, the informational search may also be automatically triggered as a latent pre-search and the informational results also queued for quick, on-demand user retrieval. [0022]: [0022] Scene camera 120 and gaze tracking camera 121 may be operated to acquire continuous real-time images that are time keyed to each other as the user views the external scene. In one embodiment, on-board computer system 118 is coupled to analyze the eye images acquired by gaze tracking camera 121 and the scene images acquired by scene camera 120 in real-time, to generate gaze direction information in real-time, to correlate the gaze direction information to the scene images in real-time, and to "lock on" and track an item in the scene images which corresponds to the item at which the user is currently gazing or staring. [0038] In a process block 420, the gazing information extracted from the eye images is used to determine which item within the external scene the user is staring or gazing directly at. For example, a gazing direction can be inferred from the gazing information, which is then used to select a localized region within the corresponding captured scene image. This localized region can then be analyzed using other techniques to lock onto a particular item at which the user may be gazing at. Various techniques such as searching for geometric shapes, contrast points, etc. may be used to determine which item within the localized region of the scene image the user is gazing at. [0050] Any of the metrics gathered by gaze tracking schemes described above may be accumulated across many users to create batched gaze tracking information. Analytical information (e.g., gazing statistics on a per user, per item, per advertisement) may then be generated based upon the batched gaze tracking information and provided to advertisers for a premium fee. The analytical information may be helpful for advertisers to refine and improve their advertisements. For example, advertisers may be interested in which specific advertisements tend to draw and hold a user's gaze and the type (e.g., emotional gaze) or intensity of their gaze based upon the pupil dilation information. Neven shows: [0036]: Gaze tracking camera 121 acquires the eye images contemporaneously with scene camera 120 acquiring the scene images and the two image streams are time keyed or synchronized with each other. [0045]: the indication of whether the user looked directly at an identified item is determined based upon both the gaze direction information and whether the user looked directly at the identified item for a threshold period of time to filter out glancing views. It should be noted that Neven shows use of a head-worn computer that can be used to gather information about the user, and Neven shows use of wearable glasses as a medium for gathering information about the user ([0021]-[0022]). Neven also shows “and an image of an eye”. Neven shows this limitation in [0065]-[0067]: where Neven shows eye tracking. Neven also shows plurality of users in [0050]: Any of the metrics gathered by gaze tracking schemes described above may be accumulated across many users to create batched gaze tracking information; and
Regarding the claim limitations below, Reference Hennessey shows:
generating a map of the environment with an indication of how many of the plurality of head-worn computers apparently looked at the object, as understood by examining the eye heading information from the plurality of head-worn computers.
Claim interpretation in light of the spec: It should be noted that this limitation is shown in applicants’ originally submitted specification at least in [0110]-[0112], [0240]. However, none of the paragraphs describe what or how the camera is disposed in line with an optical path. As such, the scope of the above claim terms is very broad and encompasses things like tracking of the eye.
In light of the specification, Neven shows the “generating … of the environment with an indication of how many of the plurality of head-worn computers apparently looked at the object, as understood by examining the eye heading information from the plurality of head-worn computers” in Fig. 4, #405-415, [0003]: receiving scene images from a head mounted gaze tracking device capturing external scenes viewed by a user wearing the head mounted device. Further, in [0017], Fig. 1 illustrates eyeglasses 102 as an example of a head mounted gaze tracking device, other types of wearable computing devices could additionally or alternatively be used. Further, in [0045]: shows each identified item may even be indexed along with location information (e.g., GPS position information) if the gaze tracking device is equipped with a GPS receiver. Further, in [0021] Gazing tracking camera 121 is positioned to acquire eye images of the user's eye.
Neven also shows plurality of users in [0050]: Any of the metrics gathered by gaze tracking schemes described above may be accumulated across many users to create batched gaze tracking information.
It should be noted though that Neven does not explicitly show “a map” as is recited in the claim. However, Hennessey shows “a map” at least in [0050]: U.S. Publication No. 2010/0295774, and entitled "Method for Automatic Mapping of Eye Tracker Data to Hypermedia Content", the contents of which are also incorporated herein by reference. [0055] Automatic calibration may be used with both 2D and 3D point of gaze estimation algorithms. For 2D scenarios, the calibration uses 2D (X,Y) key points 17. For 3D scenarios, the calibration uses 3D (X,Y,Z) key points 17. For 2D point of gaze algorithms, the calibration may include fitting coefficients to n.sup.th order polynomials which map image features to screen coordinates. For 3D point of gaze algorithms, the calibration may attempt to parametrically identify the offset between the optical axis and visual axis of the viewer. When a 3D gaze tracking system 20 is being used on a 2D display 14, the depth of the display 14 provides an additional constraint for identifying key points. [0058] In FIG. 8, numerous key points 17 are identified on web browser UI 140 displaying a web page 140. The key points 17 illustrate expected reading behavior that may be mapped to detected fixations 144 on the display 14. [0062] Referring now to FIG. 12, a sequence of key points 17 in time may be obtained for the auto-calibration procedure performed at 110 (referring back to FIG. 5) at 210, with the resulting calibration evaluated on a subset of key points 17 selected at 212, which are then used as test points. The test points are used at 214 to test the accuracy of the estimated POGs 16. Accuracy may be determined as measured by the Euclidean distance between the estimated POG 16 and the test key point 17 and the determined accuracy used to generate calibration parameters at 216. Calibration parameters (used to form the calibration profile) may include the constants for n.sup.th polynomial mapping between image features and the POG, as well as the horizontal and vertical angular offsets required to rotate the optical axis into the visual axis. [0069]: The current calibration profile for the subject 12 is obtained at 234 from the calibration profiles database 36 and the obtained calibration profile for that subject is updated at 236. The updated calibration profile includes the new constants for the polynomials that map image features to the POG, or angular offsets for rotating the optical axis into the visual axis. The calibration profile may also include information on other biometric measurements such as viewer height, size, age, gender, etc. [0083] A slightly alternate automatic calibration method takes full advantage of the pupil properties mentioned previously by, rather than calculating the user's pupil properties via a new calibration, modifying the initial calibration. It may use the same methods described previously for determining key points and accepting/rejecting said points as valid/invalid, but can use as few as two valid key points to create a mapping from initial calibration to accurate user calibration. This is because, with at least two points separate on both x and y axis, enough data is available to create the linear mapping to correct the offset and scaling between the two calibrations. [0064], [0071], [0084]. For instance, [0064]: For example, if the on-screen key point appears in the top right of the display, and the uncalibrated gaze position is found to the lower left (with respect to the center or average of the uncalibrated gaze points) then the key point is unlikely to be correlated with that gaze position estimate. After this, a similar method to the above method for cycling through calibration points can be used, with spatial constraints added. As opposed to cycling through every possible combination of key points, one can separate the points into bins based on their spatial location and order each bin based on some additional criteria (such as its approximation to the center of the spatial window). This would decrease the search space considerably. [0047]: Additional information may be used to assist in selecting the initial or default profile, such as facial recognition, or other biometric measures such as the subjects size, height, skin color, eye color, etc. [0069]: in the event that more than one subject 12 are being observed by the gaze tracking system 20, either sequentially or at the same time, the calibration profiles may be stored independently for each user. After a number of subjects 12 have been calibrated, individual viewers (for example, mom/dad/children) can be automatically identified by matching the performance of each calibration in the calibration profiles database 36 with the subject 12, and a set of current key points 17 to be used as test points.
Reference Neven and Reference Hennessey are analogous prior art to the claimed invention because the references generally relate to field of gathering user data. Further, said references are part of the same classification, i.e., G06V and G06F. Lastly, said references are filed before the effective filing date of the instant application; hence, said references are analogous prior-art references.
It would have been obvious to one of ordinary skill in the art before the effective filing date of this application for AIA to provide the teachings of Reference Hennessey, particularly the ability to map the eye gaze tracking ([0055], [0062]) and gathering and profiling “optical image data”, in the disclosure of Reference Neven, particularly in the indexing of location information for items within the field of gaze (]0045]) and use of glasses and head worn computer, for gathering user data (abstract), in order to provide for a system that integrates user data gathering through biometric sensing and user imaging as taught by Reference Hennessey (see at least in (Abstract)), where upon the execution of the method and system of Reference Hennessey for calibrating an eye gaze tracking system, the method comprising: obtaining gaze data; obtaining at least one key point corresponding to a portion of media content being displayed; linking the gaze data to the at least one key point; and generating one or more calibration parameters by comparing gaze data with associated ones of the at least one key point ([0008]) so that the process of gathering user data can be made more efficient and effective.
Further, the claimed invention is merely a combination of old elements in a similar gathering user data field of endeavor, and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that, given the existing technical ability to combine the elements as evidenced by Reference Neven in view of Reference Hennessey, the results of the combination were predictable (MPEP 2143 A).
As per claims 15, 22 and 29: Regarding the claim limitations below, Reference Hennessey shows:
wherein the map is generated using the respective locations and movement paths of each of the plurality of head-worn computers.
Claim interpretation in light of the spec: It should be noted that this limitation is shown in applicants’ originally submitted specification at least in [0110]-[0112], [0240]. However, none of the paragraphs describe what or how the camera is disposed in line with an optical path. As such, the scope of the above claim terms is very broad and encompasses things like tracking of the eye.
In light of the specification, Neven shows the “wherein … is generated using the respective locations and movement paths of each of the plurality of head-worn computers” in Fig. 4, #405-415, [0003]: receiving scene images from a head mounted gaze tracking device capturing external scenes viewed by a user wearing the head mounted device. Further, in [0017], Fig. 1 illustrates eyeglasses 102 as an example of a head mounted gaze tracking device, other types of wearable computing devices could additionally or alternatively be used. Further, in [0045]: shows each identified item may even be indexed along with location information (e.g., GPS position information) if the gaze tracking device is equipped with a GPS receiver. Further, in [0021] Gazing tracking camera 121 is positioned to acquire eye images of the user's eye.
Neven also shows plurality of users in [0050]: Any of the metrics gathered by gaze tracking schemes described above may be accumulated across many users to create batched gaze tracking information.
It should be noted though that Neven does not explicitly show “a map” as is recited in the claim. However, Hennessey shows “a map” at least in [0050]: U.S. Publication No. 2010/0295774, and entitled "Method for Automatic Mapping of Eye Tracker Data to Hypermedia Content", the contents of which are also incorporated herein by reference. [0055] Automatic calibration may be used with both 2D and 3D point of gaze estimation algorithms. For 2D scenarios, the calibration uses 2D (X,Y) key points 17. For 3D scenarios, the calibration uses 3D (X,Y,Z) key points 17. For 2D point of gaze algorithms, the calibration may include fitting coefficients to n.sup.th order polynomials which map image features to screen coordinates. For 3D point of gaze algorithms, the calibration may attempt to parametrically identify the offset between the optical axis and visual axis of the viewer. When a 3D gaze tracking system 20 is being used on a 2D display 14, the depth of the display 14 provides an additional constraint for identifying key points. [0058] In FIG. 8, numerous key points 17 are identified on web browser UI 140 displaying a web page 140. The key points 17 illustrate expected reading behavior that may be mapped to detected fixations 144 on the display 14. [0062] Referring now to FIG. 12, a sequence of key points 17 in time may be obtained for the auto-calibration procedure performed at 110 (referring back to FIG. 5) at 210, with the resulting calibration evaluated on a subset of key points 17 selected at 212, which are then used as test points. The test points are used at 214 to test the accuracy of the estimated POGs 16. Accuracy may be determined as measured by the Euclidean distance between the estimated POG 16 and the test key point 17 and the determined accuracy used to generate calibration parameters at 216. Calibration parameters (used to form the calibration profile) may include the constants for n.sup.th polynomial mapping between image features and the POG, as well as the horizontal and vertical angular offsets required to rotate the optical axis into the visual axis. [0069]: The current calibration profile for the subject 12 is obtained at 234 from the calibration profiles database 36 and the obtained calibration profile for that subject is updated at 236. The updated calibration profile includes the new constants for the polynomials that map image features to the POG, or angular offsets for rotating the optical axis into the visual axis. The calibration profile may also include information on other biometric measurements such as viewer height, size, age, gender, etc. [0083] A slightly alternate automatic calibration method takes full advantage of the pupil properties mentioned previously by, rather than calculating the user's pupil properties via a new calibration, modifying the initial calibration. It may use the same methods described previously for determining key points and accepting/rejecting said points as valid/invalid, but can use as few as two valid key points to create a mapping from initial calibration to accurate user calibration. This is because, with at least two points separate on both x and y axis, enough data is available to create the linear mapping to correct the offset and scaling between the two calibrations. [0064], [0071], [0084]. For instance, [0064]: For example, if the on-screen key point appears in the top right of the display, and the uncalibrated gaze position is found to the lower left (with respect to the center or average of the uncalibrated gaze points) then the key point is unlikely to be correlated with that gaze position estimate. After this, a similar method to the above method for cycling through calibration points can be used, with spatial constraints added. As opposed to cycling through every possible combination of key points, one can separate the points into bins based on their spatial location and order each bin based on some additional criteria (such as its approximation to the center of the spatial window). This would decrease the search space considerably. [0047]: Additional information may be used to assist in selecting the initial or default profile, such as facial recognition, or other biometric measures such as the subjects size, height, skin color, eye color, etc. [0069]: in the event that more than one subject 12 are being observed by the gaze tracking system 20, either sequentially or at the same time, the calibration profiles may be stored independently for each user. After a number of subjects 12 have been calibrated, individual viewers (for example, mom/dad/children) can be automatically identified by matching the performance of each calibration in the calibration profiles database 36 with the subject 12, and a set of current key points 17 to be used as test points.
Reference Neven and Reference Hennessey are analogous prior art to the claimed invention because the references generally relate to field of gathering user data. Further, said references are part of the same classification, i.e., G06V and G06F. Lastly, said references are filed before the effective filing date of the instant application; hence, said references are analogous prior-art references.
It would have been obvious to one of ordinary skill in the art before the effective filing date of this application for AIA to provide the teachings of Reference Hennessey, particularly the ability to map the eye gaze tracking ([0055], [0062]) and gathering and profiling “optical image data”, in the disclosure of Reference Neven, particularly in the indexing of location information for items within the field of gaze (]0045]) and use of glasses and head worn computer, for gathering user data (abstract), in order to provide for a system that integrates user data gathering through biometric sensing and user imaging as taught by Reference Hennessey (see at least in (Abstract)), where upon the execution of the method and system of Reference Hennessey for calibrating an eye gaze tracking system, the method comprising: obtaining gaze data; obtaining at least one key point corresponding to a portion of media content being displayed; linking the gaze data to the at least one key point; and generating one or more calibration parameters by comparing gaze data with associated ones of the at least one key point ([0008]) so that the process of gathering user data can be made more efficient and effective.
Further, the claimed invention is merely a combination of old elements in a similar gathering user data field of endeavor, and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that, given the existing technical ability to combine the elements as evidenced by Reference Neven in view of Reference Hennessey, the results of the combination were predictable (MPEP 2143 A).
As per claims 16, 23 and 30: Regarding the claim limitations below, Reference Hennessey shows:
wherein the map is generated using the respective locations and movement paths of each of the plurality of head-worn computers as the each of the plurality of head-worn computers moves relative to the object.
Claim interpretation in light of the spec: It should be noted that this limitation is shown in applicants’ originally submitted specification at least in [0110]-[0112], [0240]. However, none of the paragraphs describe what or how the camera is disposed in line with an optical path. As such, the scope of the above claim terms is very broad and encompasses things like tracking of the eye.
In light of the specification, Neven shows the “wherein … is generated using the respective locations and movement paths of each of the plurality of head-worn computers as the each of the plurality of head-worn computers moves relative to the object” in Fig. 4, #405-415, [0003]: receiving scene images from a head mounted gaze tracking device capturing external scenes viewed by a user wearing the head mounted device. Further, in [0017], Fig. 1 illustrates eyeglasses 102 as an example of a head mounted gaze tracking device, other types of wearable computing devices could additionally or alternatively be used. Further, in [0045]: shows each identified item may even be indexed along with location information (e.g., GPS position information) if the gaze tracking device is equipped with a GPS receiver. Further, in [0021] Gazing tracking camera 121 is positioned to acquire eye images of the user's eye.
Neven also shows plurality of users in [0050]: Any of the metrics gathered by gaze tracking schemes described above may be accumulated across many users to create batched gaze tracking information.
It should be noted though that Neven does not explicitly show “a map” as is recited in the claim. However, Hennessey shows “a map” at least in [0050]: U.S. Publication No. 2010/0295774, and entitled "Method for Automatic Mapping of Eye Tracker Data to Hypermedia Content", the contents of which are also incorporated herein by reference. [0055] Automatic calibration may be used with both 2D and 3D point of gaze estimation algorithms. For 2D scenarios, the calibration uses 2D (X,Y) key points 17. For 3D scenarios, the calibration uses 3D (X,Y,Z) key points 17. For 2D point of gaze algorithms, the calibration may include fitting coefficients to n.sup.th order polynomials which map image features to screen coordinates. For 3D point of gaze algorithms, the calibration may attempt to parametrically identify the offset between the optical axis and visual axis of the viewer. When a 3D gaze tracking system 20 is being used on a 2D display 14, the depth of the display 14 provides an additional constraint for identifying key points. [0058] In FIG. 8, numerous key points 17 are identified on web browser UI 140 displaying a web page 140. The key points 17 illustrate expected reading behavior that may be mapped to detected fixations 144 on the display 14. [0062] Referring now to FIG. 12, a sequence of key points 17 in time may be obtained for the auto-calibration procedure performed at 110 (referring back to FIG. 5) at 210, with the resulting calibration evaluated on a subset of key points 17 selected at 212, which are then used as test points. The test points are used at 214 to test the accuracy of the estimated POGs 16. Accuracy may be determined as measured by the Euclidean distance between the estimated POG 16 and the test key point 17 and the determined accuracy used to generate calibration parameters at 216. Calibration parameters (used to form the calibration profile) may include the constants for n.sup.th polynomial mapping between image features and the POG, as well as the horizontal and vertical angular offsets required to rotate the optical axis into the visual axis. [0069]: The current calibration profile for the subject 12 is obtained at 234 from the calibration profiles database 36 and the obtained calibration profile for that subject is updated at 236. The updated calibration profile includes the new constants for the polynomials that map image features to the POG, or angular offsets for rotating the optical axis into the visual axis. The calibration profile may also include information on other biometric measurements such as viewer height, size, age, gender, etc. [0083] A slightly alternate automatic calibration method takes full advantage of the pupil properties mentioned previously by, rather than calculating the user's pupil properties via a new calibration, modifying the initial calibration. It may use the same methods described previously for determining key points and accepting/rejecting said points as valid/invalid, but can use as few as two valid key points to create a mapping from initial calibration to accurate user calibration. This is because, with at least two points separate on both x and y axis, enough data is available to create the linear mapping to correct the offset and scaling between the two calibrations. [0064], [0071], [0084]. For instance, [0064]: For example, if the on-screen key point appears in the top right of the display, and the uncalibrated gaze position is found to the lower left (with respect to the center or average of the uncalibrated gaze points) then the key point is unlikely to be correlated with that gaze position estimate. After this, a similar method to the above method for cycling through calibration points can be used, with spatial constraints added. As opposed to cycling through every possible combination of key points, one can separate the points into bins based on their spatial location and order each bin based on some additional criteria (such as its approximation to the center of the spatial window). This would decrease the search space considerably. [0047]: Additional information may be used to assist in selecting the initial or default profile, such as facial recognition, or other biometric measures such as the subjects size, height, skin color, eye color, etc. [0069]: in the event that more than one subject 12 are being observed by the gaze tracking system 20, either sequentially or at the same time, the calibration profiles may be stored independently for each user. After a number of subjects 12 have been calibrated, individual viewers (for example, mom/dad/children) can be automatically identified by matching the performance of each calibration in the calibration profiles database 36 with the subject 12, and a set of current key points 17 to be used as test points.
Reference Neven and Reference Hennessey are analogous prior art to the claimed invention because the references generally relate to field of gathering user data. Further, said references are part of the same classification, i.e., G06V and G06F. Lastly, said references are filed before the effective filing date of the instant application; hence, said references are analogous prior-art references.
It would have been obvious to one of ordinary skill in the art before the effective filing date of this application for AIA to provide the teachings of Reference Hennessey, particularly the ability to map the eye gaze tracking ([0055], [0062]) and gathering and profiling “optical image data”, in the disclosure of Reference Neven, particularly in the indexing of location information for items within the field of gaze (]0045]) and use of glasses and head worn computer, for gathering user data (abstract), in order to provide for a system that integrates user data gathering through biometric sensing and user imaging as taught by Reference Hennessey (see at least in (Abstract)), where upon the execution of the method and system of Reference Hennessey for calibrating an eye gaze tracking system, the method comprising: obtaining gaze data; obtaining at least one key point corresponding to a portion of media content being displayed; linking the gaze data to the at least one key point; and generating one or more calibration parameters by comparing gaze data with associated ones of the at least one key point ([0008]) so that the process of gathering user data can be made more efficient and effective.
Further, the claimed invention is merely a combination of old elements in a similar gathering user data field of endeavor, and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that, given the existing technical ability to combine the elements as evidenced by Reference Neven in view of Reference Hennessey, the results of the combination were predictable (MPEP 2143 A).
As per claims 17, 24 and 31: Regarding the claim limitations below, Reference Hennessey shows:
wherein the map comprises a second indication of the eye heading information.
Neven shows the above limitation in Fig. 4, #405-415, [0003]: receiving scene images from a head mounted gaze tracking device capturing external scenes viewed by a user wearing the head mounted device. Further, in [0017], Fig. 1 illustrates eyeglasses 102 as an example of a head mounted gaze tracking device, other types of wearable computing devices could additionally or alternatively be used. Further, in [0045]: shows each identified item may even be indexed along with location information (e.g., GPS position information) if the gaze tracking device is equipped with a GPS receiver. Further, in [0021] Gazing tracking camera 121 is positioned to acquire eye images of the user's eye. Also see, [0040], Fig. 3 #175: Gazing Log. Further, in [0021] Gazing tracking camera 121 is positioned to acquire eye images of the user's eye. Also see, [0040], Fig. 3 #175: Gazing Log. [0048]: For example, when a user looks directly at an identified item being tracked by the system, the user may trigger an information search (e.g., Wikipedia search, Google search, etc.) to obtain general or specific background information on the identified item being tracked by signaling with a particular eye gesture or via finger-operable touch pads 124, 126 on device 100. The information search results can then be quickly served up to the user in the heads-up display (e.g., lens elements 110 and 112). Thus, latent pre-searching can improve visual searching to identify an item by triggering the visual search when it first enters the user's peripheral view before it reaches his direct gaze and speed up potential informational searches on identified items submitted by the user. In one embodiment, once a latent pre-search has identified a particular item, the informational search may also be automatically triggered as a latent pre-search and the informational results also queued for quick, on-demand user retrieval. [0045]: In one embodiment, the indication of whether the user looked directly at an identified item is determined based upon both the gaze direction information and whether the user looked directly at the identified item for a threshold period of time to filter out glancing views. In one embodiment, each identified item may even be indexed along with location information (e.g., GPS position information) if the gaze tracking device is equipped with a GPS receiver. Of course, position information may also be acquired from local cell towers or WiFi hot spot locations at the time that the gaze tracking device transmits the scene images. Gazing log 175 may be generated and stored locally at server system 160 or archived into a database 170. [0048]: For example, when a user looks directly at an identified item being tracked by the system, the user may trigger an information search (e.g., Wikipedia search, Google search, etc.) to obtain general or specific background information on the identified item being tracked by signaling with a particular eye gesture or via finger-operable touch pads 124, 126 on device 100. The information search results can then be quickly served up to the user in the heads-up display (e.g., lens elements 110 and 112). Thus, latent pre-searching can improve visual searching to identify an item by triggering the visual search when it first enters the user's peripheral view before it reaches his direct gaze and speed up potential informational searches on identified items submitted by the user. In one embodiment, once a latent pre-search has identified a particular item, the informational search may also be automatically triggered as a latent pre-search and the informational results also queued for quick, on-demand user retrieval. [0022]: [0022] Scene camera 120 and gaze tracking camera 121 may be operated to acquire continuous real-time images that are time keyed to each other as the user views the external scene. In one embodiment, on-board computer system 118 is coupled to analyze the eye images acquired by gaze tracking camera 121 and the scene images acquired by scene camera 120 in real-time, to generate gaze direction information in real-time, to correlate the gaze direction information to the scene images in real-time, and to "lock on" and track an item in the scene images which corresponds to the item at which the user is currently gazing or staring. [0038] In a process block 420, the gazing information extracted from the eye images is used to determine which item within the external scene the user is staring or gazing directly at. For example, a gazing direction can be inferred from the gazing information, which is then used to select a localized region within the corresponding captured scene image. This localized region can then be analyzed using other techniques to lock onto a particular item at which the user may be gazing at. Various techniques such as searching for geometric shapes, contrast points, etc. may be used to determine which item within the localized region of the scene image the user is gazing at. [0050] Any of the metrics gathered by gaze tracking schemes described above may be accumulated across many users to create batched gaze tracking information. Analytical information (e.g., gazing statistics on a per user, per item, per advertisement) may then be generated based upon the batched gaze tracking information and provided to advertisers for a premium fee. The analytical information may be helpful for advertisers to refine and improve their advertisements. For example, advertisers may be interested in which specific advertisements tend to draw and hold a user's gaze and the type (e.g., emotional gaze) or intensity of their gaze based upon the pupil dilation information. Neven shows: [0036]: Gaze tracking camera 121 acquires the eye images contemporaneously with scene camera 120 acquiring the scene images and the two image streams are time keyed or synchronized with each other. [0045]: the indication of whether the user looked directly at an identified item is determined based upon both the gaze direction information and whether the user looked directly at the identified item for a threshold period of time to filter out glancing views. It should be noted that Neven shows use of a head-worn computer that can be used to gather information about the user, and Neven shows use of wearable glasses as a medium for gathering information about the user ([0021]-[0022]). Neven also shows “and an image of an eye”. Neven shows this limitation in [0065]-[0067]: where Neven shows eye tracking. Neven also shows plurality of users in [0050]: Any of the metrics gathered by gaze tracking schemes described above may be accumulated across many users to create batched gaze tracking information.
It should be noted though that Neven does not explicitly show “a map” as is recited in the claim. However, Hennessey shows “a map” at least in [0050]: U.S. Publication No. 2010/0295774, and entitled "Method for Automatic Mapping of Eye Tracker Data to Hypermedia Content", the contents of which are also incorporated herein by reference. [0055] Automatic calibration may be used with both 2D and 3D point of gaze estimation algorithms. For 2D scenarios, the calibration uses 2D (X,Y) key points 17. For 3D scenarios, the calibration uses 3D (X,Y,Z) key points 17. For 2D point of gaze algorithms, the calibration may include fitting coefficients to n.sup.th order polynomials which map image features to screen coordinates. For 3D point of gaze algorithms, the calibration may attempt to parametrically identify the offset between the optical axis and visual axis of the viewer. When a 3D gaze tracking system 20 is being used on a 2D display 14, the depth of the display 14 provides an additional constraint for identifying key points. [0058] In FIG. 8, numerous key points 17 are identified on web browser UI 140 displaying a web page 140. The key points 17 illustrate expected reading behavior that may be mapped to detected fixations 144 on the display 14. [0062] Referring now to FIG. 12, a sequence of key points 17 in time may be obtained for the auto-calibration procedure performed at 110 (referring back to FIG. 5) at 210, with the resulting calibration evaluated on a subset of key points 17 selected at 212, which are then used as test points. The test points are used at 214 to test the accuracy of the estimated POGs 16. Accuracy may be determined as measured by the Euclidean distance between the estimated POG 16 and the test key point 17 and the determined accuracy used to generate calibration parameters at 216. Calibration parameters (used to form the calibration profile) may include the constants for n.sup.th polynomial mapping between image features and the POG, as well as the horizontal and vertical angular offsets required to rotate the optical axis into the visual axis. [0069]: The current calibration profile for the subject 12 is obtained at 234 from the calibration profiles database 36 and the obtained calibration profile for that subject is updated at 236. The updated calibration profile includes the new constants for the polynomials that map image features to the POG, or angular offsets for rotating the optical axis into the visual axis. The calibration profile may also include information on other biometric measurements such as viewer height, size, age, gender, etc. [0083] A slightly alternate automatic calibration method takes full advantage of the pupil properties mentioned previously by, rather than calculating the user's pupil properties via a new calibration, modifying the initial calibration. It may use the same methods described previously for determining key points and accepting/rejecting said points as valid/invalid, but can use as few as two valid key points to create a mapping from initial calibration to accurate user calibration. This is because, with at least two points separate on both x and y axis, enough data is available to create the linear mapping to correct the offset and scaling between the two calibrations. [0064], [0071], [0084]. For instance, [0064]: For example, if the on-screen key point appears in the top right of the display, and the uncalibrated gaze position is found to the lower left (with respect to the center or average of the uncalibrated gaze points) then the key point is unlikely to be correlated with that gaze position estimate. After this, a similar method to the above method for cycling through calibration points can be used, with spatial constraints added. As opposed to cycling through every possible combination of key points, one can separate the points into bins based on their spatial location and order each bin based on some additional criteria (such as its approximation to the center of the spatial window). This would decrease the search space considerably. [0047]: Additional information may be used to assist in selecting the initial or default profile, such as facial recognition, or other biometric measures such as the subjects size, height, skin color, eye color, etc. [0069]: in the event that more than one subject 12 are being observed by the gaze tracking system 20, either sequentially or at the same time, the calibration profiles may be stored independently for each user. After a number of subjects 12 have been calibrated, individual viewers (for example, mom/dad/children) can be automatically identified by matching the performance of each calibration in the calibration profiles database 36 with the subject 12, and a set of current key points 17 to be used as test points.
Reference Neven and Reference Hennessey are analogous prior art to the claimed invention because the references generally relate to field of gathering user data. Further, said references are part of the same classification, i.e., G06V and G06F. Lastly, said references are filed before the effective filing date of the instant application; hence, said references are analogous prior-art references.
It would have been obvious to one of ordinary skill in the art before the effective filing date of this application for AIA to provide the teachings of Reference Hennessey, particularly the ability to map the eye gaze tracking ([0055], [0062]) and gathering and profiling “optical image data”, in the disclosure of Reference Neven, particularly in the indexing of location information for items within the field of gaze (]0045]) and use of glasses and head worn computer, for gathering user data (abstract), in order to provide for a system that integrates user data gathering through biometric sensing and user imaging as taught by Reference Hennessey (see at least in (Abstract)), where upon the execution of the method and system of Reference Hennessey for calibrating an eye gaze tracking system, the method comprising: obtaining gaze data; obtaining at least one key point corresponding to a portion of media content being displayed; linking the gaze data to the at least one key point; and generating one or more calibration parameters by comparing gaze data with associated ones of the at least one key point ([0008]) so that the process of gathering user data can be made more efficient and effective.
Further, the claimed invention is merely a combination of old elements in a similar gathering user data field of endeavor, and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that, given the existing technical ability to combine the elements as evidenced by Reference Neven in view of Reference Hennessey, the results of the combination were predictable (MPEP 2143 A).
As per claims 18, 25 and 32: Regarding the claim limitations below, Reference Hennessey shows:
wherein the map comprises a third indication of respective locations of each of the plurality of head-worn computers when the object was in respective field of views of each of the plurality of head-worn computers.
Neven shows the above limitation in Fig. 4, #405-415, [0003]: receiving scene images from a head mounted gaze tracking device capturing external scenes viewed by a user wearing the head mounted device. Further, in [0017], Fig. 1 illustrates eyeglasses 102 as an example of a head mounted gaze tracking device, other types of wearable computing devices could additionally or alternatively be used. Further, in [0045]: shows each identified item may even be indexed along with location information (e.g., GPS position information) if the gaze tracking device is equipped with a GPS receiver. Further, in [0021] Gazing tracking camera 121 is positioned to acquire eye images of the user's eye. Also see, [0040], Fig. 3 #175: Gazing Log. Further, in [0021] Gazing tracking camera 121 is positioned to acquire eye images of the user's eye. Also see, [0040], Fig. 3 #175: Gazing Log. [0048]: For example, when a user looks directly at an identified item being tracked by the system, the user may trigger an information search (e.g., Wikipedia search, Google search, etc.) to obtain general or specific background information on the identified item being tracked by signaling with a particular eye gesture or via finger-operable touch pads 124, 126 on device 100. The information search results can then be quickly served up to the user in the heads-up display (e.g., lens elements 110 and 112). Thus, latent pre-searching can improve visual searching to identify an item by triggering the visual search when it first enters the user's peripheral view before it reaches his direct gaze and speed up potential informational searches on identified items submitted by the user. In one embodiment, once a latent pre-search has identified a particular item, the informational search may also be automatically triggered as a latent pre-search and the informational results also queued for quick, on-demand user retrieval. [0045]: In one embodiment, the indication of whether the user looked directly at an identified item is determined based upon both the gaze direction information and whether the user looked directly at the identified item for a threshold period of time to filter out glancing views. In one embodiment, each identified item may even be indexed along with location information (e.g., GPS position information) if the gaze tracking device is equipped with a GPS receiver. Of course, position information may also be acquired from local cell towers or WiFi hot spot locations at the time that the gaze tracking device transmits the scene images. Gazing log 175 may be generated and stored locally at server system 160 or archived into a database 170. [0048]: For example, when a user looks directly at an identified item being tracked by the system, the user may trigger an information search (e.g., Wikipedia search, Google search, etc.) to obtain general or specific background information on the identified item being tracked by signaling with a particular eye gesture or via finger-operable touch pads 124, 126 on device 100. The information search results can then be quickly served up to the user in the heads-up display (e.g., lens elements 110 and 112). Thus, latent pre-searching can improve visual searching to identify an item by triggering the visual search when it first enters the user's peripheral view before it reaches his direct gaze and speed up potential informational searches on identified items submitted by the user. In one embodiment, once a latent pre-search has identified a particular item, the informational search may also be automatically triggered as a latent pre-search and the informational results also queued for quick, on-demand user retrieval. [0022]: [0022] Scene camera 120 and gaze tracking camera 121 may be operated to acquire continuous real-time images that are time keyed to each other as the user views the external scene. In one embodiment, on-board computer system 118 is coupled to analyze the eye images acquired by gaze tracking camera 121 and the scene images acquired by scene camera 120 in real-time, to generate gaze direction information in real-time, to correlate the gaze direction information to the scene images in real-time, and to "lock on" and track an item in the scene images which corresponds to the item at which the user is currently gazing or staring. [0038] In a process block 420, the gazing information extracted from the eye images is used to determine which item within the external scene the user is staring or gazing directly at. For example, a gazing direction can be inferred from the gazing information, which is then used to select a localized region within the corresponding captured scene image. This localized region can then be analyzed using other techniques to lock onto a particular item at which the user may be gazing at. Various techniques such as searching for geometric shapes, contrast points, etc. may be used to determine which item within the localized region of the scene image the user is gazing at. [0050] Any of the metrics gathered by gaze tracking schemes described above may be accumulated across many users to create batched gaze tracking information. Analytical information (e.g., gazing statistics on a per user, per item, per advertisement) may then be generated based upon the batched gaze tracking information and provided to advertisers for a premium fee. The analytical information may be helpful for advertisers to refine and improve their advertisements. For example, advertisers may be interested in which specific advertisements tend to draw and hold a user's gaze and the type (e.g., emotional gaze) or intensity of their gaze based upon the pupil dilation information. Neven shows: [0036]: Gaze tracking camera 121 acquires the eye images contemporaneously with scene camera 120 acquiring the scene images and the two image streams are time keyed or synchronized with each other. [0045]: the indication of whether the user looked directly at an identified item is determined based upon both the gaze direction information and whether the user looked directly at the identified item for a threshold period of time to filter out glancing views. It should be noted that Neven shows use of a head-worn computer that can be used to gather information about the user, and Neven shows use of wearable glasses as a medium for gathering information about the user ([0021]-[0022]). Neven also shows “and an image of an eye”. Neven shows this limitation in [0065]-[0067]: where Neven shows eye tracking. Neven also shows plurality of users in [0050]: Any of the metrics gathered by gaze tracking schemes described above may be accumulated across many users to create batched gaze tracking information.
It should be noted though that Neven does not explicitly show “a map” as is recited in the claim. However, Hennessey shows “a map” at least in [0050]: U.S. Publication No. 2010/0295774, and entitled "Method for Automatic Mapping of Eye Tracker Data to Hypermedia Content", the contents of which are also incorporated herein by reference. [0055] Automatic calibration may be used with both 2D and 3D point of gaze estimation algorithms. For 2D scenarios, the calibration uses 2D (X,Y) key points 17. For 3D scenarios, the calibration uses 3D (X,Y,Z) key points 17. For 2D point of gaze algorithms, the calibration may include fitting coefficients to n.sup.th order polynomials which map image features to screen coordinates. For 3D point of gaze algorithms, the calibration may attempt to parametrically identify the offset between the optical axis and visual axis of the viewer. When a 3D gaze tracking system 20 is being used on a 2D display 14, the depth of the display 14 provides an additional constraint for identifying key points. [0058] In FIG. 8, numerous key points 17 are identified on web browser UI 140 displaying a web page 140. The key points 17 illustrate expected reading behavior that may be mapped to detected fixations 144 on the display 14. [0062] Referring now to FIG. 12, a sequence of key points 17 in time may be obtained for the auto-calibration procedure performed at 110 (referring back to FIG. 5) at 210, with the resulting calibration evaluated on a subset of key points 17 selected at 212, which are then used as test points. The test points are used at 214 to test the accuracy of the estimated POGs 16. Accuracy may be determined as measured by the Euclidean distance between the estimated POG 16 and the test key point 17 and the determined accuracy used to generate calibration parameters at 216. Calibration parameters (used to form the calibration profile) may include the constants for n.sup.th polynomial mapping between image features and the POG, as well as the horizontal and vertical angular offsets required to rotate the optical axis into the visual axis. [0069]: The current calibration profile for the subject 12 is obtained at 234 from the calibration profiles database 36 and the obtained calibration profile for that subject is updated at 236. The updated calibration profile includes the new constants for the polynomials that map image features to the POG, or angular offsets for rotating the optical axis into the visual axis. The calibration profile may also include information on other biometric measurements such as viewer height, size, age, gender, etc. [0083] A slightly alternate automatic calibration method takes full advantage of the pupil properties mentioned previously by, rather than calculating the user's pupil properties via a new calibration, modifying the initial calibration. It may use the same methods described previously for determining key points and accepting/rejecting said points as valid/invalid, but can use as few as two valid key points to create a mapping from initial calibration to accurate user calibration. This is because, with at least two points separate on both x and y axis, enough data is available to create the linear mapping to correct the offset and scaling between the two calibrations. [0064], [0071], [0084]. For instance, [0064]: For example, if the on-screen key point appears in the top right of the display, and the uncalibrated gaze position is found to the lower left (with respect to the center or average of the uncalibrated gaze points) then the key point is unlikely to be correlated with that gaze position estimate. After this, a similar method to the above method for cycling through calibration points can be used, with spatial constraints added. As opposed to cycling through every possible combination of key points, one can separate the points into bins based on their spatial location and order each bin based on some additional criteria (such as its approximation to the center of the spatial window). This would decrease the search space considerably. [0047]: Additional information may be used to assist in selecting the initial or default profile, such as facial recognition, or other biometric measures such as the subjects size, height, skin color, eye color, etc. [0069]: in the event that more than one subject 12 are being observed by the gaze tracking system 20, either sequentially or at the same time, the calibration profiles may be stored independently for each user. After a number of subjects 12 have been calibrated, individual viewers (for example, mom/dad/children) can be automatically identified by matching the performance of each calibration in the calibration profiles database 36 with the subject 12, and a set of current key points 17 to be used as test points.
Reference Neven and Reference Hennessey are analogous prior art to the claimed invention because the references generally relate to field of gathering user data. Further, said references are part of the same classification, i.e., G06V and G06F. Lastly, said references are filed before the effective filing date of the instant application; hence, said references are analogous prior-art references.
It would have been obvious to one of ordinary skill in the art before the effective filing date of this application for AIA to provide the teachings of Reference Hennessey, particularly the ability to map the eye gaze tracking ([0055], [0062]) and gathering and profiling “optical image data”, in the disclosure of Reference Neven, particularly in the indexing of location information for items within the field of gaze (]0045]) and use of glasses and head worn computer, for gathering user data (abstract), in order to provide for a system that integrates user data gathering through biometric sensing and user imaging as taught by Reference Hennessey (see at least in (Abstract)), where upon the execution of the method and system of Reference Hennessey for calibrating an eye gaze tracking system, the method comprising: obtaining gaze data; obtaining at least one key point corresponding to a portion of media content being displayed; linking the gaze data to the at least one key point; and generating one or more calibration parameters by comparing gaze data with associated ones of the at least one key point ([0008]) so that the process of gathering user data can be made more efficient and effective.
Further, the claimed invention is merely a combination of old elements in a similar gathering user data field of endeavor, and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that, given the existing technical ability to combine the elements as evidenced by Reference Neven in view of Reference Hennessey, the results of the combination were predictable (MPEP 2143 A).
As per claims 19 and 26: Regarding the claim limitations below, Reference Hennessey shows:
wherein the map comprises a fourth indication of respective lengths of time the object was in respective field of views of each of the plurality of head-worn computers.
Neven shows the above limitation in Fig. 4, #405-415, [0003]: receiving scene images from a head mounted gaze tracking device capturing external scenes viewed by a user wearing the head mounted device. Further, in [0017], Fig. 1 illustrates eyeglasses 102 as an example of a head mounted gaze tracking device, other types of wearable computing devices could additionally or alternatively be used. Further, in [0045]: shows each identified item may even be indexed along with location information (e.g., GPS position information) if the gaze tracking device is equipped with a GPS receiver. Further, in [0021] Gazing tracking camera 121 is positioned to acquire eye images of the user's eye. Also see, [0040], Fig. 3 #175: Gazing Log. Further, in [0021] Gazing tracking camera 121 is positioned to acquire eye images of the user's eye. Also see, [0040], Fig. 3 #175: Gazing Log. [0048]: For example, when a user looks directly at an identified item being tracked by the system, the user may trigger an information search (e.g., Wikipedia search, Google search, etc.) to obtain general or specific background information on the identified item being tracked by signaling with a particular eye gesture or via finger-operable touch pads 124, 126 on device 100. The information search results can then be quickly served up to the user in the heads-up display (e.g., lens elements 110 and 112). Thus, latent pre-searching can improve visual searching to identify an item by triggering the visual search when it first enters the user's peripheral view before it reaches his direct gaze and speed up potential informational searches on identified items submitted by the user. In one embodiment, once a latent pre-search has identified a particular item, the informational search may also be automatically triggered as a latent pre-search and the informational results also queued for quick, on-demand user retrieval. [0045]: In one embodiment, the indication of whether the user looked directly at an identified item is determined based upon both the gaze direction information and whether the user looked directly at the identified item for a threshold period of time to filter out glancing views. In one embodiment, each identified item may even be indexed along with location information (e.g., GPS position information) if the gaze tracking device is equipped with a GPS receiver. Of course, position information may also be acquired from local cell towers or WiFi hot spot locations at the time that the gaze tracking device transmits the scene images. Gazing log 175 may be generated and stored locally at server system 160 or archived into a database 170. [0048]: For example, when a user looks directly at an identified item being tracked by the system, the user may trigger an information search (e.g., Wikipedia search, Google search, etc.) to obtain general or specific background information on the identified item being tracked by signaling with a particular eye gesture or via finger-operable touch pads 124, 126 on device 100. The information search results can then be quickly served up to the user in the heads-up display (e.g., lens elements 110 and 112). Thus, latent pre-searching can improve visual searching to identify an item by triggering the visual search when it first enters the user's peripheral view before it reaches his direct gaze and speed up potential informational searches on identified items submitted by the user. In one embodiment, once a latent pre-search has identified a particular item, the informational search may also be automatically triggered as a latent pre-search and the informational results also queued for quick, on-demand user retrieval. [0022]: [0022] Scene camera 120 and gaze tracking camera 121 may be operated to acquire continuous real-time images that are time keyed to each other as the user views the external scene. In one embodiment, on-board computer system 118 is coupled to analyze the eye images acquired by gaze tracking camera 121 and the scene images acquired by scene camera 120 in real-time, to generate gaze direction information in real-time, to correlate the gaze direction information to the scene images in real-time, and to "lock on" and track an item in the scene images which corresponds to the item at which the user is currently gazing or staring. [0038] In a process block 420, the gazing information extracted from the eye images is used to determine which item within the external scene the user is staring or gazing directly at. For example, a gazing direction can be inferred from the gazing information, which is then used to select a localized region within the corresponding captured scene image. This localized region can then be analyzed using other techniques to lock onto a particular item at which the user may be gazing at. Various techniques such as searching for geometric shapes, contrast points, etc. may be used to determine which item within the localized region of the scene image the user is gazing at. [0050] Any of the metrics gathered by gaze tracking schemes described above may be accumulated across many users to create batched gaze tracking information. Analytical information (e.g., gazing statistics on a per user, per item, per advertisement) may then be generated based upon the batched gaze tracking information and provided to advertisers for a premium fee. The analytical information may be helpful for advertisers to refine and improve their advertisements. For example, advertisers may be interested in which specific advertisements tend to draw and hold a user's gaze and the type (e.g., emotional gaze) or intensity of their gaze based upon the pupil dilation information. Neven shows: [0036]: Gaze tracking camera 121 acquires the eye images contemporaneously with scene camera 120 acquiring the scene images and the two image streams are time keyed or synchronized with each other. [0045]: the indication of whether the user looked directly at an identified item is determined based upon both the gaze direction information and whether the user looked directly at the identified item for a threshold period of time to filter out glancing views. It should be noted that Neven shows use of a head-worn computer that can be used to gather information about the user, and Neven shows use of wearable glasses as a medium for gathering information about the user ([0021]-[0022]). Neven also shows “and an image of an eye”. Neven shows this limitation in [0065]-[0067]: where Neven shows eye tracking. Neven also shows plurality of users in [0050]: Any of the metrics gathered by gaze tracking schemes described above may be accumulated across many users to create batched gaze tracking information.
It should be noted though that Neven does not explicitly show “a map” as is recited in the claim. However, Hennessey shows “a map” at least in [0050]: U.S. Publication No. 2010/0295774, and entitled "Method for Automatic Mapping of Eye Tracker Data to Hypermedia Content", the contents of which are also incorporated herein by reference. [0055] Automatic calibration may be used with both 2D and 3D point of gaze estimation algorithms. For 2D scenarios, the calibration uses 2D (X,Y) key points 17. For 3D scenarios, the calibration uses 3D (X,Y,Z) key points 17. For 2D point of gaze algorithms, the calibration may include fitting coefficients to n.sup.th order polynomials which map image features to screen coordinates. For 3D point of gaze algorithms, the calibration may attempt to parametrically identify the offset between the optical axis and visual axis of the viewer. When a 3D gaze tracking system 20 is being used on a 2D display 14, the depth of the display 14 provides an additional constraint for identifying key points. [0058] In FIG. 8, numerous key points 17 are identified on web browser UI 140 displaying a web page 140. The key points 17 illustrate expected reading behavior that may be mapped to detected fixations 144 on the display 14. [0062] Referring now to FIG. 12, a sequence of key points 17 in time may be obtained for the auto-calibration procedure performed at 110 (referring back to FIG. 5) at 210, with the resulting calibration evaluated on a subset of key points 17 selected at 212, which are then used as test points. The test points are used at 214 to test the accuracy of the estimated POGs 16. Accuracy may be determined as measured by the Euclidean distance between the estimated POG 16 and the test key point 17 and the determined accuracy used to generate calibration parameters at 216. Calibration parameters (used to form the calibration profile) may include the constants for n.sup.th polynomial mapping between image features and the POG, as well as the horizontal and vertical angular offsets required to rotate the optical axis into the visual axis. [0069]: The current calibration profile for the subject 12 is obtained at 234 from the calibration profiles database 36 and the obtained calibration profile for that subject is updated at 236. The updated calibration profile includes the new constants for the polynomials that map image features to the POG, or angular offsets for rotating the optical axis into the visual axis. The calibration profile may also include information on other biometric measurements such as viewer height, size, age, gender, etc. [0083] A slightly alternate automatic calibration method takes full advantage of the pupil properties mentioned previously by, rather than calculating the user's pupil properties via a new calibration, modifying the initial calibration. It may use the same methods described previously for determining key points and accepting/rejecting said points as valid/invalid, but can use as few as two valid key points to create a mapping from initial calibration to accurate user calibration. This is because, with at least two points separate on both x and y axis, enough data is available to create the linear mapping to correct the offset and scaling between the two calibrations. [0064], [0071], [0084]. For instance, [0064]: For example, if the on-screen key point appears in the top right of the display, and the uncalibrated gaze position is found to the lower left (with respect to the center or average of the uncalibrated gaze points) then the key point is unlikely to be correlated with that gaze position estimate. After this, a similar method to the above method for cycling through calibration points can be used, with spatial constraints added. As opposed to cycling through every possible combination of key points, one can separate the points into bins based on their spatial location and order each bin based on some additional criteria (such as its approximation to the center of the spatial window). This would decrease the search space considerably. [0047]: Additional information may be used to assist in selecting the initial or default profile, such as facial recognition, or other biometric measures such as the subjects size, height, skin color, eye color, etc. [0069]: in the event that more than one subject 12 are being observed by the gaze tracking system 20, either sequentially or at the same time, the calibration profiles may be stored independently for each user. After a number of subjects 12 have been calibrated, individual viewers (for example, mom/dad/children) can be automatically identified by matching the performance of each calibration in the calibration profiles database 36 with the subject 12, and a set of current key points 17 to be used as test points.
Reference Neven and Reference Hennessey are analogous prior art to the claimed invention because the references generally relate to field of gathering user data. Further, said references are part of the same classification, i.e., G06V and G06F. Lastly, said references are filed before the effective filing date of the instant application; hence, said references are analogous prior-art references.
It would have been obvious to one of ordinary skill in the art before the effective filing date of this application for AIA to provide the teachings of Reference Hennessey, particularly the ability to map the eye gaze tracking ([0055], [0062]) and gathering and profiling “optical image data”, in the disclosure of Reference Neven, particularly in the indexing of location information for items within the field of gaze (]0045]) and use of glasses and head worn computer, for gathering user data (abstract), in order to provide for a system that integrates user data gathering through biometric sensing and user imaging as taught by Reference Hennessey (see at least in (Abstract)), where upon the execution of the method and system of Reference Hennessey for calibrating an eye gaze tracking system, the method comprising: obtaining gaze data; obtaining at least one key point corresponding to a portion of media content being displayed; linking the gaze data to the at least one key point; and generating one or more calibration parameters by comparing gaze data with associated ones of the at least one key point ([0008]) so that the process of gathering user data can be made more efficient and effective.
Further, the claimed invention is merely a combination of old elements in a similar gathering user data field of endeavor, and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that, given the existing technical ability to combine the elements as evidenced by Reference Neven in view of Reference Hennessey, the results of the combination were predictable (MPEP 2143 A).
As per claims 20 and 27: Regarding the claim limitations below, Reference Hennessey shows:
wherein the map comprises a fifth indication of respective movements of each of the plurality of head-worn computers after the object exits respective field of views of each of the plurality of head-worn computers.
Neven shows the above limitation in Fig. 4, #405-415, [0003]: receiving scene images from a head mounted gaze tracking device capturing external scenes viewed by a user wearing the head mounted device. Further, in [0017], Fig. 1 illustrates eyeglasses 102 as an example of a head mounted gaze tracking device, other types of wearable computing devices could additionally or alternatively be used. Further, in [0045]: shows each identified item may even be indexed along with location information (e.g., GPS position information) if the gaze tracking device is equipped with a GPS receiver. Further, in [0021] Gazing tracking camera 121 is positioned to acquire eye images of the user's eye. Also see, [0040], Fig. 3 #175: Gazing Log. Further, in [0021] Gazing tracking camera 121 is positioned to acquire eye images of the user's eye. Also see, [0040], Fig. 3 #175: Gazing Log. [0048]: For example, when a user looks directly at an identified item being tracked by the system, the user may trigger an information search (e.g., Wikipedia search, Google search, etc.) to obtain general or specific background information on the identified item being tracked by signaling with a particular eye gesture or via finger-operable touch pads 124, 126 on device 100. The information search results can then be quickly served up to the user in the heads-up display (e.g., lens elements 110 and 112). Thus, latent pre-searching can improve visual searching to identify an item by triggering the visual search when it first enters the user's peripheral view before it reaches his direct gaze and speed up potential informational searches on identified items submitted by the user. In one embodiment, once a latent pre-search has identified a particular item, the informational search may also be automatically triggered as a latent pre-search and the informational results also queued for quick, on-demand user retrieval. [0045]: In one embodiment, the indication of whether the user looked directly at an identified item is determined based upon both the gaze direction information and whether the user looked directly at the identified item for a threshold period of time to filter out glancing views. In one embodiment, each identified item may even be indexed along with location information (e.g., GPS position information) if the gaze tracking device is equipped with a GPS receiver. Of course, position information may also be acquired from local cell towers or WiFi hot spot locations at the time that the gaze tracking device transmits the scene images. Gazing log 175 may be generated and stored locally at server system 160 or archived into a database 170. [0048]: For example, when a user looks directly at an identified item being tracked by the system, the user may trigger an information search (e.g., Wikipedia search, Google search, etc.) to obtain general or specific background information on the identified item being tracked by signaling with a particular eye gesture or via finger-operable touch pads 124, 126 on device 100. The information search results can then be quickly served up to the user in the heads-up display (e.g., lens elements 110 and 112). Thus, latent pre-searching can improve visual searching to identify an item by triggering the visual search when it first enters the user's peripheral view before it reaches his direct gaze and speed up potential informational searches on identified items submitted by the user. In one embodiment, once a latent pre-search has identified a particular item, the informational search may also be automatically triggered as a latent pre-search and the informational results also queued for quick, on-demand user retrieval. [0022]: [0022] Scene camera 120 and gaze tracking camera 121 may be operated to acquire continuous real-time images that are time keyed to each other as the user views the external scene. In one embodiment, on-board computer system 118 is coupled to analyze the eye images acquired by gaze tracking camera 121 and the scene images acquired by scene camera 120 in real-time, to generate gaze direction information in real-time, to correlate the gaze direction information to the scene images in real-time, and to "lock on" and track an item in the scene images which corresponds to the item at which the user is currently gazing or staring. [0038] In a process block 420, the gazing information extracted from the eye images is used to determine which item within the external scene the user is staring or gazing directly at. For example, a gazing direction can be inferred from the gazing information, which is then used to select a localized region within the corresponding captured scene image. This localized region can then be analyzed using other techniques to lock onto a particular item at which the user may be gazing at. Various techniques such as searching for geometric shapes, contrast points, etc. may be used to determine which item within the localized region of the scene image the user is gazing at. [0050] Any of the metrics gathered by gaze tracking schemes described above may be accumulated across many users to create batched gaze tracking information. Analytical information (e.g., gazing statistics on a per user, per item, per advertisement) may then be generated based upon the batched gaze tracking information and provided to advertisers for a premium fee. The analytical information may be helpful for advertisers to refine and improve their advertisements. For example, advertisers may be interested in which specific advertisements tend to draw and hold a user's gaze and the type (e.g., emotional gaze) or intensity of their gaze based upon the pupil dilation information. Neven shows: [0036]: Gaze tracking camera 121 acquires the eye images contemporaneously with scene camera 120 acquiring the scene images and the two image streams are time keyed or synchronized with each other. [0045]: the indication of whether the user looked directly at an identified item is determined based upon both the gaze direction information and whether the user looked directly at the identified item for a threshold period of time to filter out glancing views. It should be noted that Neven shows use of a head-worn computer that can be used to gather information about the user, and Neven shows use of wearable glasses as a medium for gathering information about the user ([0021]-[0022]). Neven also shows “and an image of an eye”. Neven shows this limitation in [0065]-[0067]: where Neven shows eye tracking. Neven also shows plurality of users in [0050]: Any of the metrics gathered by gaze tracking schemes described above may be accumulated across many users to create batched gaze tracking information.
It should be noted though that Neven does not explicitly show “a map” as is recited in the claim. However, Hennessey shows “a map” at least in [0050]: U.S. Publication No. 2010/0295774, and entitled "Method for Automatic Mapping of Eye Tracker Data to Hypermedia Content", the contents of which are also incorporated herein by reference. [0055] Automatic calibration may be used with both 2D and 3D point of gaze estimation algorithms. For 2D scenarios, the calibration uses 2D (X,Y) key points 17. For 3D scenarios, the calibration uses 3D (X,Y,Z) key points 17. For 2D point of gaze algorithms, the calibration may include fitting coefficients to n.sup.th order polynomials which map image features to screen coordinates. For 3D point of gaze algorithms, the calibration may attempt to parametrically identify the offset between the optical axis and visual axis of the viewer. When a 3D gaze tracking system 20 is being used on a 2D display 14, the depth of the display 14 provides an additional constraint for identifying key points. [0058] In FIG. 8, numerous key points 17 are identified on web browser UI 140 displaying a web page 140. The key points 17 illustrate expected reading behavior that may be mapped to detected fixations 144 on the display 14. [0062] Referring now to FIG. 12, a sequence of key points 17 in time may be obtained for the auto-calibration procedure performed at 110 (referring back to FIG. 5) at 210, with the resulting calibration evaluated on a subset of key points 17 selected at 212, which are then used as test points. The test points are used at 214 to test the accuracy of the estimated POGs 16. Accuracy may be determined as measured by the Euclidean distance between the estimated POG 16 and the test key point 17 and the determined accuracy used to generate calibration parameters at 216. Calibration parameters (used to form the calibration profile) may include the constants for n.sup.th polynomial mapping between image features and the POG, as well as the horizontal and vertical angular offsets required to rotate the optical axis into the visual axis. [0069]: The current calibration profile for the subject 12 is obtained at 234 from the calibration profiles database 36 and the obtained calibration profile for that subject is updated at 236. The updated calibration profile includes the new constants for the polynomials that map image features to the POG, or angular offsets for rotating the optical axis into the visual axis. The calibration profile may also include information on other biometric measurements such as viewer height, size, age, gender, etc. [0083] A slightly alternate automatic calibration method takes full advantage of the pupil properties mentioned previously by, rather than calculating the user's pupil properties via a new calibration, modifying the initial calibration. It may use the same methods described previously for determining key points and accepting/rejecting said points as valid/invalid, but can use as few as two valid key points to create a mapping from initial calibration to accurate user calibration. This is because, with at least two points separate on both x and y axis, enough data is available to create the linear mapping to correct the offset and scaling between the two calibrations. [0064], [0071], [0084]. For instance, [0064]: For example, if the on-screen key point appears in the top right of the display, and the uncalibrated gaze position is found to the lower left (with respect to the center or average of the uncalibrated gaze points) then the key point is unlikely to be correlated with that gaze position estimate. After this, a similar method to the above method for cycling through calibration points can be used, with spatial constraints added. As opposed to cycling through every possible combination of key points, one can separate the points into bins based on their spatial location and order each bin based on some additional criteria (such as its approximation to the center of the spatial window). This would decrease the search space considerably. [0047]: Additional information may be used to assist in selecting the initial or default profile, such as facial recognition, or other biometric measures such as the subjects size, height, skin color, eye color, etc. [0069]: in the event that more than one subject 12 are being observed by the gaze tracking system 20, either sequentially or at the same time, the calibration profiles may be stored independently for each user. After a number of subjects 12 have been calibrated, individual viewers (for example, mom/dad/children) can be automatically identified by matching the performance of each calibration in the calibration profiles database 36 with the subject 12, and a set of current key points 17 to be used as test points.
Reference Neven and Reference Hennessey are analogous prior art to the claimed invention because the references generally relate to field of gathering user data. Further, said references are part of the same classification, i.e., G06V and G06F. Lastly, said references are filed before the effective filing date of the instant application; hence, said references are analogous prior-art references.
It would have been obvious to one of ordinary skill in the art before the effective filing date of this application for AIA to provide the teachings of Reference Hennessey, particularly the ability to map the eye gaze tracking ([0055], [0062]) and gathering and profiling “optical image data”, in the disclosure of Reference Neven, particularly in the indexing of location information for items within the field of gaze (]0045]) and use of glasses and head worn computer, for gathering user data (abstract), in order to provide for a system that integrates user data gathering through biometric sensing and user imaging as taught by Reference Hennessey (see at least in (Abstract)), where upon the execution of the method and system of Reference Hennessey for calibrating an eye gaze tracking system, the method comprising: obtaining gaze data; obtaining at least one key point corresponding to a portion of media content being displayed; linking the gaze data to the at least one key point; and generating one or more calibration parameters by comparing gaze data with associated ones of the at least one key point ([0008]) so that the process of gathering user data can be made more efficient and effective.
Further, the claimed invention is merely a combination of old elements in a similar gathering user data field of endeavor, and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that, given the existing technical ability to combine the elements as evidenced by Reference Neven in view of Reference Hennessey, the results of the combination were predictable (MPEP 2143 A).
As per claim(s) 33:
Examiner notes that claim(s) 33 recites substantially similar limitations as those addressed above in claim(s) 19 and 20, respectively. Therefore, claim(s) 33 are rejected for the same reasons as set forth above for claim(s) 19 and 20, respectively.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
NPL Reference:
S. M. Kim, M. Sked and Q. Ji, "Non-intrusive eye gaze tracking under natural head movements," The 26th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, 2004, pp. 2271-2274, doi: 10.1109/IEMBS.2004.1403660.
This reference is concerned with eye gaze tracking system under natural head movements. The system consists of one CCD camera and two mirrors. Based on geometric and linear algebra calculations, the mirrors rotate to follow head movements in order to keep the eyes within the view of the camera. Our system allows the subjects head to move 30 cm horizontally and 20 cm vertically, with spatial gaze resolutions about 6 degree and 7 degree, respectively and a frame rate about 10 Hz. We also introduce a hierarchical generalized regression neural networks (H-GRNN) scheme to map eye and mirror parameters to gaze, achieving a gaze estimation accuracy of 92% under head movements. The use of H-GRNN also eliminates the need for personal calibration for new subjects since H-GRNN can generalize. Preliminary experiments show our system is accurate and robust in gaze tracking under large head movements.
Saffo, Paul. "Google Glass signals a wearables revolution" cnn.com. April 16, 2014.
https://www.cnn.com/2014/04/16/opinion/saffo-google-glass/index.html
This reference is concerned with how the arrival of Google Glass has resulted in a debate over where and when info-glasses can be worn. Just like similar debates over pagers, cell phones and smartphones in years past, wearables will likely be everywhere. Besides, unlike smartphones, info-glass hardware is going to quickly shrink into near-invisibility. Within a few years, smart glasses will be indistinguishable from an ordinary pair of vintage 2014 specs. And after that? How about info-contact lens that can check your vital signs? And privacy? Forget about it. We are destined to become like tagged bears, constantly tracked, but too addicted to the data stream to switch our intimate devices off.
Harris, Stacy. "Pivothead Wearable Imaging Launches Indiegogo Campaign for SMART Eyewear" businesswire.com. November 12, 2013.
https://www.businesswire.com/news/home/20131112006678/en/Pivothead-Wearable-Imaging-Launches-Indiegogo-Campaign-for-SMART-Eyewear
This reference is concerned Pivothead SMART provides the full 1080p high-definition (HD) video and 8 megapixel photo capture capability of Pivothead’s award-winning first generation eyewear, plus Smart Mods™ - a revolutionary new design approach to wearable technology. The new modular SMART lineup packs a multitude of new features, including full HD wireless live streaming, and is geared towards effortless real-time sharing activities across numerous applications.
Foreign Reference:
CN 107783291 A. LI Y et al. The device has frame (10) which is provided to wear head-mounted display device to user's head. An optical assembly (40) is connected to image processor (30) for displaying two-dimensional image slice. A liquid lens (50) is arranged on side of optical assembly arranged near the user's eye, so that the two-dimensional image slices displayed on optical assembly are imaged at corresponding depth of field, and light field information of three-dimensional object is input into human eye.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NANCY PRASAD whose telephone number is (571)270-3265. The examiner can normally be reached M-F: 8:00 AM - 4:30 PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Patricia Munson can be reached at (571)270-5396. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/N.N.P/Examiner, Art Unit 3624 /PATRICIA H MUNSON/Supervisory Patent Examiner, Art Unit 3624
