DETAILED ACTION
Notice of AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
Claim 25 is rejected under 35 U.S.C. 102(a)(1) as being anticipated by Murai.
Regarding Claim 25, Murai discloses “A non-transitory computer-readable medium having instructions encoded thereon that, when executed by a processing device, cause the processing device to” (Figure 1, Item 6 ‘computing apparatus’, and Paragraph [0042], Lines 8 – 12 (Notice that a non-transitory computer readable medium of the onboard memory of the CPU processing device has computer programs or instructions that execute to perform actions that follow.)), “receive at least one image” (Figure 1, Item 10 ‘imaging apparatus’, and Paragraph [0046], Lines 1 – 4 (Notice that the marker information obtaining unit 14 implemented as a computer program performs receiving an image from digital imaging apparatus 10.)), “define, within the at least one image, (i) a first region in relation to a user, wherein the first region corresponds to a first window” (Figure 1, Paragraph [0046], Lines 1 – 4, and Figure 5(6) (Notice that the marker information obtaining unit analyzes an incoming image to define a first region in relation the user’s fingers (markers), where the first region is an outer first window as shown in Figure 5(6).)), “and (ii) a second region, the second region corresponding to a second window” (Figure 1, Paragraph [0046], Lines 1 – 4, and Figure 5(6) (Notice that the marker information obtaining unit analyzes an incoming image to define a second region in relation the user’s fingers(markers), where the second region is an inner first window 24 as shown in Figure 5(6).)), “process the at least one image to identify (a) information corresponding to a hand gesture performed by a user within the second region, and (b) information corresponding to the second window” (Figure 1, Items 14, 16, Paragraph [0046], and Figure 5(6) (Notice the processing by marker information obtaining unit 14 and marker generating unit 16 allow identification of information the corresponds the a finger formation gesture of user within the inner first window 24 as the second region and location information corresponding to the inner first window 24 as the second region.)), “and provide an instruction corresponding to the identified hand gesture with respect to the second window” (Figure 1, Item 20 ‘target object operating unit’, and Paragraph [0051], Lines 1 – 3, and Figure 5(6) (Notice that an instruction is provided according to the finger formation gesture to expand window 24 as shown in Figure 5(6).)).
Claim 26 is rejected under 35 U.S.C. 102(a)(1) as being anticipated by Katz.
Regarding Claim 26, Katz discloses “A system comprising: at least one processor” (Figure 1, Items 2 ‘system’, 6 ‘processor’, and Page 12, Line 25), “configured to: receive at least one image” (Page 13, Line 15), “received one or more audio inputs” (Page 15, Lines 4 – 6), “process the at least one image to identify (a) information corresponding to a line of sight of user directed towards a device” (Page 14, Lines 25 – 28 ‘object the user is looking at’ or in other words the lines of sight of user directed toward a activatable object in front of the display such that the user is directing sight towards the display 8.)), “(b) information corresponding to gesture of a hand of the user towards a location depicted in the at least one image” (Page 14, Lines 1 – 24 (Notice that gesture of a hand is identified with respect to an activatable object at a location. In other words, a location of an activatable object depicted in the at least one image is related to the activating object.)), “process the one or more audio inputs to identify a command” (Page 15, Lines 4 – 6), “format content based on one or more characteristics of a surface” (Figure 3, Item ‘d’, and Page 13, Lines 1 - 3 (Notice that the content of an activatable object is formatted as a 3D image based on the characteristic of the distance of the surface from the plane of display device 8.)), “wherein the surface does not comprise the hand of the user” (Page 14, Lines 1 - 24 (Notice that surface of an activatable object does not comprise the hand of the user.)), “and provide to the device, (a) one or more instructions corresponding to the identified command” (Page 14, Line 29 – Page 15, Line 6 (Notice that a message or command associated with the displayed activatable object is recalled from memory that can depend on both an activating gesture and voice command with regard to the displayed activatable object recognized by an eye gaze or line of sight to the surface/space of the activatable object (Page 14, Lines 25 – 28)), “and (b) the content as formatted based on the one or more characteristics of the surface and the line of sight of the user in relation to the location” (Page 15, Lines 18 – 21 (Notice that the resizing of the activatable object is displayed in the physical 2D/3D space/surface in response to a hand gesture, where resizing of the activatable object occurs in the physical 2D/3D space/surface that is occupied by the activatable object that is recognized by eye gaze or eye direction (Page 14, Lines 25 – 28).)).
Claim Rejections - 35 USC § 103
Claims 1 – 2, 4, 6 – 7, 10 – 11, 14, 16, 18, and 22-23 are rejected under 35 U.S.C. 103 as being unpatentable over Katz in view of Cunningham.
Regarding Claim 1, Katz discloses “A system comprising: at least one processor” (Figure 1, Items 2 ‘system’, 6 ‘processor’, and Page 12, Line 25), “configured to: receive at least one image” (Page 13, Line 15), “process the at least one image to identify (a), within the received at least one image, information corresponding to a gesture performed by a hand of a user” (Page 14, Lines 1 – 24 (Notice that gesture of hand is identified with respect to an activatable object, where the gesture is obtained from a received image.)), “and (b) information corresponding to a physical surface depicted in the at least one image” (Page 13, Lines 23 – 27 (Notice that processed image identifies information about the location of activating object with respect to a activatable object in the physical 2D/3D space/surface. In other words, a physical plane surface of an activatable object depicted in the at least one image is related to the location of the activating object.)), “wherein the physical surface does not comprise the hand of the user” (Page 13, Lines 23 – 27 (Notice that said physical plane surface of an activatable object does not comprise the hand of the user.)), “format content based on one or more characteristics of the identified physical surface” (Figure 3, Item ‘d’, and Page 13, Lines 1 - 3 (Notice that the content of an activatable object is formatted as a 3D image based on the characteristic of the distance of the physical plane surface from the plane of display device 8.)), “and display content as formatted based on the one or more characteristics of the physical surface depicted in the at least one image and as associated with the identified gesture performed by the hand of the user [ ] in relation to an identified eye gaze of the user in relation to the information corresponding to the physical surface” (Page 15, Lines 18 – 21 (Notice that the resizing of the activatable object (formatted as a 3D image based on the characteristic of the distance of the physical plane surface from the plane of display device 8) is displayed in the physical 2D/3D space/surface in response to a gesture of a hand, where resizing of the activatable object occurs in the physical 2D/3D space/surface that is occupied by the activatable object that is recognized by eye gaze or eye direction (Page 14, Lines 25 – 28). In other words, the eye gaze or eye direction has a relationship with information, where the information is the physical plane surface of an activatable object depicted in that at least one image). Furthermore, Katz teaches wherein to display content the at least one processor is further configured to display content associated with the identified hand gesture and an identified voice command (Page 14, Line 29 – Page 15, Line 6 (Notice that a message associated with the displayed activatable object is recalled from memory that can depend on both an activating gesture and voice command with regard to the displayed activatable object.)). Also, Katz does not explicitly disclose “and [a] determined viewing ray”, but teaches that gaze (which is associated with a view ray) can be used to recognize an activatable object (Page 14, Lines 25 – 27). However, Cunningham teaches tracking of user’s eye gaze by mapping eye gaze measurements from two dimensional images to a two-dimension coordinate space of display (Paragraph [0078], Lines 8 - 17, where one of ordinary skill in the art would recognize a line intersecting a gaze in the two-dimensional images to the two-dimensional coordinate space would be of ray from the eye location to gaze target location.
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to display content the at least one processor is further configured to display content associated with the identified hand gesture and “an identified voice command, and a determined viewing ray” because one having ordinary skill in the art would want to provide plural input conditions which must be met to prevent accidental selection of content.
Regarding Claim 2, Katz and Cunningham, the combination of hereinafter referenced as KC, disclose/teach everything claimed as applied above (See Claim 1). In addition, Katz discloses “wherein to process the at least one image the at least one processor is further configured to identify information corresponding to an eye gaze of the user” (Page 14, Lines 25 – 28 ‘object the user is looking at’ or in other words the object the user is gazing at with his/her eyes.)).
Regarding Claim 4, KC disclose/teach everything claimed as applied above (See Claim 2). In addition, Katz discloses “wherein the surface comprises a display” (Figure 1, Item 16 (Notice the display of the activatable object in the 2D/3D space/surface of 3D display 8.)), “and wherein the at least one processor is further configured to define a first region of the display based on the eye gaze” (Page 14, Lines 25 – 28 (Notice that the activatable object the user is looking at in the 2D/3D space/surface of 3D display 8 defines or is the region that the activatable object occupies.
Regarding Claim 6, KC disclose/teach everything claimed as applied above (See Claim 4). In addition, Katz fails to explicitly disclose “wherein the at least one processor is further configured to define a second region of the display based on a change in the eye gaze; and wherein to display content the at least one processor is further configured to position a cursor within the second region”. However, Cunningham teaches tracking of user’s eye gaze to find cursor position on a projected display to make selections in a user interface (Paragraph [0078], Lines 1 – 17).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to provide “wherein the at least one processor is further configured to define a second region of the display based on a change in the eye gaze; and wherein to display content the at least one processor is further configured to position [a] cursor within the second region” because one having ordinary skill in the art would want to move around a display with changing eye positions and make cursor selections in a user interface.
Regarding Claim 7, KC disclose/ teach everything claimed as applied above (See Claim 2). In addition, Katz fails to explicitly disclose “wherein to process the at least one image the at least one processor is further configured to determine a viewing ray with respect to the eye of the user and the surface”. However, Cunningham teaches tracking of user’s eye gaze by mapping eye gaze measurements from two dimensional images to a two-dimension coordinate space of display (Paragraph [0078], Lines 8 - 17, where one of ordinary skill in the art would recognize a line intersecting a gaze in the two-dimensional images to the two-dimensional coordinate space would be of ray from the eye location to gaze target/element location.
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to provide “wherein to process the at least one image the at least one processor is further configured to determine a viewing ray with respect to the eye of the user and the surface” because one having ordinary skill in the art would want determine the directionality of the eyes from a facial plane to a viewing target plane.
Regarding Claim 10, KC disclose/teach everything claimed as applied above (See Claim 2). In addition, Katz discloses “wherein to process the at least one image the at least one processor is further configured to define within the at least one image a first region in relation to the user” (Figure 3 and Page 17, Lines 9 – 19 (Notice that the cone generated outward toward an activatable object defines a first region in relation to the user in the plane of the viewed activatable object for object activation)).
Regarding Claim 11, KC disclose/ teach everything claimed as applied above (See Claim 10). In addition, Katz discloses “wherein to process the at least one image the at least one processor is further configured to identify a presence of a pointing element within the first region” (Figure 3 and Page 17, Lines 9 – 19 (Notice that user finger in the defined first region as explained above is identified as a pointing element with the defined first region.)). Also, Katz fails to explicitly disclose “wherein to display content the at least one processor is further configured to display a cursor on the surface at a location that corresponds to the presence of the pointing element within the first region”. However, Cunningham teaches tracking of user’s eye gaze to find cursor position on a projected display to make selections in a user interface (Paragraph [0078], Lines 1 – 17).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to provide “wherein to display content the at least one processor is further configured to display a cursor on the surface at a location that corresponds to the presence of the pointing element within the first region” because one having ordinary skill in the art would want to provide selection feedback via a cursor like that for an eye gaze selection to prevent unintentional input.
Regarding Claim 14, KC disclose/teach everything claimed as applied above (See Claim 10). In addition, Katz discloses “wherein the first region corresponds to a first interface displayed on the surface (Figure 3 and Page 17, Lines 9 – 19 (Notice that the cone generated outward toward a activatable object defines a first region in relation to the user in the plane surface of the viewed activatable object for object activation, such that the defined first region is a user interfacing region.)), “and wherein to process the at least one image the at least one processor is further configured to identify the hand gesture within the first region” (Page 17, Lines 16 – 18 ‘predetermined gesture’), “and wherein to display content the at least one processor is further configured to provide an instruction corresponding the hand gesture with respect to the first interface” (Page 17, Lines 18 – 19 (Notice that an instruction to activate the activatable object is issued in response to the predetermined gesture in the user interfacing region with the activatable object displayed.)).
Regarding Claim 16, KC disclose/teach everything claimed as applied above (See Claim 1). In addition, Katz discloses “wherein to identify information corresponding to a surface the at least one processor is further configured to identify, in the at least one image, a surface associated with the identified hand gesture” (Figure 3 and Page 17, Lines 9 – 19 (Notice that the cone generated outward toward a activatable object defines a first region in relation to the user gesture that identifies a plane or surface of the viewed activatable object)), “and wherein to display content the at least one processor is further configured to display the content associated with the identified hand gesture in relation to the identified surface (Page 15, Lines 18 – 21 (Notice that object is displayed in the 2D/3D space/surface or plane in response a hand gesture.)).
Regarding Claim 18, KC disclose/teach everything claimed as applied above (See Claim 1). In addition, Katz discloses “wherein to display the content is to display the content in relation to the surface based on one or more characteristics of the surface (Figure 3, Item ‘d’ (Notice that the content of an activatable object is displayed according the characteristic distance of the surface.)).
Regarding Claim 22, KC disclose/teach everything claimed as applied above (See Claim 1). In addition, Katz discloses “wherein the at least one processor is further configured to adjust one or more setting associated with an illumination device based on the identified hand gesture” (Page 15, Lines 18 – 21 (Notice that resizing an object that is displayed in the 2D/3D space/surface in response a hand gesture requires the illumination device (display device 8) to adjust at least one setting to cause the resizing based upon the hand gesture.))
Regarding Claim 23, KC disclose/teach everything claimed as applied above (See Claim 1). In addition, Katz discloses “wherein the at least one processor is further configured to: receive one or more audio inputs and process the one or more audio inputs to identify a command” (Page 15, Lines 4 – 6), “and wherein to display content the at least one processor is further configured to display content associated with the identified hand gesture and the identified command in relation to the surface (Page 14, Line 29 – Page 15, Line 6 (Notice that a message associated with the displayed activatable object is recalled from memory that can depend on both an activating gesture and voice command with regard to the displayed activatable object.)).
Claim 3 is rejected under 35 U.S.C. 103 as being unpatentable over Katz, in view of Cunningham, and further in view of Yamazaki.
Regarding Claim 3, KC disclose/teach everything claimed as applied above (See Claim 2). In addition, Katz and Cunningham fail to explicitly disclose “wherein to process the at least one image the at least one processor is further configured to identify information corresponding to a pupil of the eye of the user in relation to one or more areas of a face of the user”.
In similar field of endeavor, Yamazaki teaches detection of a user’s gaze via a relative positional relationship between the eye portion and the pupil portions, where the eye portion is on area of the face of user (Paragraph [0030], Lines 1 – 7).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to provide “wherein to process the at least one image the at least one processor is further configured to identify information corresponding to a pupil of the eye of the user in relation to one or more areas of a face of the user” because one having ordinary skill in the art would want utilize a relative position that is in the same frame of reference of a pupil to prevent additional spatial calculations.
Claims 12, 13, and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Katz, in view of Cunningham, and further in view of Hennessey.
Regarding Claim 12, KC disclose/teach everything claimed as applied above (See Claim 10). In addition, Katz and Cunningham fail to explicitly disclose “wherein to define a first region the at least one processor is further configured to define a second region within the first region”.
In similar field of endeavor, Hennessey teaches the defining of key points 17 (i.e. a first region) that are locations of content (objects) and the defining of point of gaze regions 16 (i.e. a second region), where the point of gaze regions 16 are within a key point 17 (Figure 1, Item 16, 17 and Paragraph [0061], Lines 1 – 12).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to provide “wherein to define a first region the at least one processor is further configured to define a second region within the first region” because one having ordinary skill in the art would want to ensure a point of gaze correctly falls on an activatable object as a key point.
Regarding Claim 13, Katz, Cunningham, and Hennessey, the combination of hereinafter referenced as KCH disclose/ teach everything claimed as applied above (See Claim 12). In addition, Katz discloses “wherein to process the at least one image the at least one processor is further configured to identify a presence of a pointing element within [a] region” (Figure 3 and Page 17, Lines 9 – 19 (Notice that user finger in the defined first region as explained above is identified as a pointing element with the defined first region.)). Also, Katz and Cunningham fail to explicitly disclose “the second region” and “wherein to display content the at least one processor is further configured to display a cursor on the [physical surface] at a location that corresponds to the presence of the pointing element within the second region”. However, Hennessy teaches the defining of key points 17 (i.e. a first region) that are locations of content (objects/elements) and the defining of point of gaze regions 16 (i.e. a second region), where the point of gaze regions 16 are within a key point 17 (Figure 1, Item 16, 17 and Paragraph [0061], Lines 1 – 12) and Cunningham teaches tracking of user’s eye gaze to find cursor position on a projected display to make selections in a user interface (Paragraph [0078], Lines 1 – 17).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to provide “the second region” and “wherein to display content the at least one processor is further configured to display a cursor on the surface at a location that corresponds to the presence of the pointing element within the second region” because one having ordinary skill in the art would want to provide selection feedback via a cursor like that for an eye gaze selection to prevent unintentional input via finer granularity.
Regarding Claim 15, KCH disclose/ teach everything claimed as applied above (See Claim 14). In addition, Katz discloses “wherein to define a first region that at least one processor is further configured to define a second region” (Page 14, Lines 22 – 24 (Notice that first and second activatable objects with activation regions can be set.)), “the second region corresponding to the a second interface displayed on the surface (Page 14, Lines 22 – 24 (Notice that the user interfacing region for the second region can be set in a surface, but KC does not explicitly disclose the same surface.)), “and wherein to process the at least one image the at least one processor is further configured to identify the hand gesture within the second region” (Page 14, Lines 22 – 24 ‘different gestures’), “and wherein to display content the at least one processor is further configured to provide an instruction corresponding the hand gesture with respect to the second interface” (Page 14, Lines 22 - 24 (Notice that an instruction to activate an activatable object is issued in response to the predetermined gesture in the second user interfacing region with an activatable object displayed.)). Although KC fail to explicitly disclose “on the surface”, Hennessey teaches user interface elements 174 in a same surface or layer of smart phone 170 in Figure 10.
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to provide “on the surface because one having ordinary skill in the art would want to provide selection feedback via different activation object (e.g. hands) without twisting one’s body.
Response to Amendments and Arguments
Applicants amendments and arguments filed September 22, 2025 have been fully considered.
First, the Examiner disagrees that the amendment to Claim 1 has overcome the 35 U.S.C. 103 rejections of Claims 1-2, 4, 6 – 7, 10 – 11, 14, 16, 18, and 22 – 23 as set forth and made of record in the Office Action mailed March 21, 2025. The Examiner disagrees because of the reasoning set forth in the rejections of Claims 1 – 2, 4, 6 – 7, 10 – 11, 14, 16, 18, and 22 – 23 above.
Second, the Examiner disagrees that - - The Office Action (p. 3) now alleges that cited portions of Cunningham teach of suggest the above-highlighted feature(s) - - (REMARKS, Page 8, Lines 16 – 17 (line reference made by all written lines, including headings and cited portions, but excluding page headings)). The Examiner disagrees because Cunningham has only been replied upon to teach tracking of user’s eye gaze by mapping eye gaze measurements from two dimensional images to a two-dimension coordinate space of display (Paragraph [0078], Lines 8 – 17), where one of ordinary skill in the art would recognize a line intersecting a gaze in the two-dimensional images to the two-dimensional coordinate space would be of ray from the eye location to gaze target location with regard to “a determined viewing ray”.
Third, the Examiner disagrees that - - the Office action concedes, cited portions of Katz are not relied upon to teach these features - - (REMARKS, Page 9, Lines 12 – 13 (line reference made by all written lines, including headings and cited portions, but excluding page headings). The Examiner disagrees because Katz is replied upon to disclose or teach all features except “and [a] determined viewing ray” (see the rejection of Claim 1 as made of record in the Office Action mailed March 21, 2025).
Finally, the Examiner attempted to schedule an applicant initiated interview on October 8, 2025 as requested in the “REMARKS”, but was unable to secure a telephone connection to the contact number listed in the “REMARKS” filed September 22, 2025. The Examiner is open to scheduling an interview via at least a direct telephone request, via an Automated Interview Request (AIR), or via an alternative telephone number provided in following official responses.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BRIAN M BUTCHER whose telephone number is (571)270-5575. The examiner can normally be reached on Monday – Friday from 6:30 AM to 3:00 PM.
If attempts to reach the examiner by telephone are unsuccessful, the examiner's supervisor, Ke Xiao, can be reached at (571) 272 - 7776. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
/BRIAN M BUTCHER/Primary Examiner, Art Unit 2627 October 11, 2025