DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Election/Restrictions
Applicant’s election without traverse of claims 1-2, 7-9, 14-16, and 20 in the reply filed on 01/22/2026 is acknowledged.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-2, 7-9, 14-16, and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 2017/0092007 by Goldberg et al. in view of US 11,398,042 by Zhao.
Regarding claim 1, Goldberg et al. discloses a method for collecting line-of-sight direction data, comprising:
collecting an object image of a target object through a first camera, and determining an object three-dimensional coordinate of the target object under a first camera coordinate system based on the object image, wherein the target object is an object at which a user stares (fig. 3, paragraph 0138 teaches “The next step 302 is video capture, wherein digital cameras or sensors in the smart eyeglasses capture images of the user's field of view.”, paragraph 0163 teaches “Each frame of the captured video field is time stamped. The user's view is eye-tracked and the moment when the user's gaze is determined to show an interest in something is time-stamped. The system uses the coordinates of the user's eye gaze and the time stamp to find the frame(s) in the video field matching that time stamp and then identifying the pixels matching the coordinates of the eye gaze. Once the pixels are identified, they are subjected to the video processing techniques described above to create super-pixels.”);
collecting a facial image of the user through a second depth camera, and determining an eye three-dimensional coordinate of the user under a second camera coordinate system based on the facial image (in addition to discussion above, paragraph 0137 teaches “Referring to FIG. 3, the first step 301 involves tracking the eye movement of the user wearing the smart eyeglasses. Eye tracking is used to determine the object(s) that the user is looking at in a defined visual field”, paragraph 0126 teaches “The smart eyeglasses 200 further comprise cameras/sensors 202(B) for tracking eye movements. In some embodiments, the system may employ one or more inward facing digital cameras or sensors, which are used to track the movement of the eyes and to determine the foveal and peripheral fields of focus. This information helps to determine the object(s) that a user may be looking at.”, paragraph 0163 teaches “In an embodiment, tracking the eye of a user generates coordinate data that defines a coordinate system for the user's visual field. A captured video field is mapped to the visual field of the user, ensuring that the system displays the image or video of the same object or scene to the user, which the user appears to be interested in as determined by the eye tracking step.”, paragraph 0146-0147);
determining an object coordinate of the object three-dimensional coordinate and an eye coordinate of the eye three-dimensional coordinate under each collection camera coordinate system (in addition to discussion above, paragraph 0139 teaches “Next the captured video field is mapped to the visual field of the user, as shown in step 303. This step ensures that the system displays the image or video of the same object or scene to the user, which the user appears to be interested in as determined by the eye tracking step.”, paragraph 0165 teaches “FIG. 4 is a flowchart illustrating a method of mapping a captured video field to a user's visual field, according to one embodiment. Referring to FIG. 4, in the first step 401 eye tracking data and video capture data are time synced. As mentioned above, eye tracking data marks where the person is looking in a defined visual field, wherein the size of the defined visual field is, for example, X×Y pixels. When eye tracking data indicates a person is interested in a particular object, the corresponding time stamped video is retrieved, as shown in step 402. The area or object of interest within the defined visual field may be denoted as X′×Y′ pixels. It may be noted that X′×Y′ is a smaller subset of pixels of the defined visual field, and could be as small as 100×100 pixels. Next in step 403, the coordinates of a person's eye focus are translated from the visual field to the captured video field. In this step, the system maps where the person is looking in the visual field to the video field. Accordingly, X′,Y′ in the visual field is translated to X″, Y″ in the video field. Thus, the coordinates of one or more locations from the user's visual field are translated to the coordinate system of the video field to yield video field coordinates defining at least one, and preferably a plurality of objects of interest in the video field.”); and
determining line-of-sight direction data of the user based on the object coordinate and the eye coordinate under each collection camera coordinate system (in addition to discussion above, paragraph 0122 teaches “In some embodiments, the present specification discloses the use of a visual interface, such as eyeglasses, that are capable of performing eye tracking functions, capturing video, mapping the visual field to the captured video field, displaying the identified captured video field to the user and enabling the user to control that display, and, finally, visual field sharing.”, paragraph 0123 teaches “In some embodiments, the methods and devices of the present specification may use a high definition camera to capture a person's entire visual field in great detail. In some embodiments, the methods and devices of the present specification will employ eye tracking to determine where a person is looking and then map that location to a video field. Once mapped to the video field, it will retrieve that portion of the video field and allow a person to zoom in, pan around, and manipulate the resultant “enhanced” image accordingly.”, paragraph 0151 teaches “In the eye-tracking system, the visual field of the test subject is recorded using a first camera (76) rigidly connected to the head (80) of the test subject so that it faces forward and is recorded in a visual field video, the movement of the pupils of the test subject is recorded with a second camera (77), which is also rigidly connected to the head (80) of the test subject, and is recorded in an eye video, and the eye video and the visual field video (9) are recorded on a video system and time-synchronized, wherein for each individual image of the eye video, therefore for each eye image (78) the pupil coordinates xa,ya are determined, the correlation function K between pupil coordinates xa,ya on the eye video and coordinates xb,yb of the corresponding point of vision B, i.e. the point the test subject fixes on, on which the visual field image (79) of the visual field video (9) is determined, and after determining the correlation function K for each individual image from the pupil coordinates xa,ya on the eye video, the coordinates xb,yb of the corresponding point of vision B on the visual field video are extrapolated, wherein to determine the pupil coordinates xa,ya for each individual image of the eye video with a visual detection program, the contrasts of the pupils to the surroundings are automatically recorded, all points of the individual image, which are darker than a predefined degree of darkness, are identified, these points record and limit an area of darkness corresponding to the pupil and the focus of the area of darkness, which corresponds to the middle of the pupil with the pupil coordinates xa,ya, is determined”).
Goldberg et al. fails to disclose first depth camera, second depth camera.
Zhao discloses (Abstract teaches “The method includes capturing a plurality of first depth images by a first image capturing device comprising one or more first depth cameras from a plurality of positions at a same time; transmitting the first depth images to a central computing device in real time; integrating the plurality of first depth images as one frame, based on calibration parameters of the one or more first depth cameras;”)
It would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to incorporate the ability to include first depth camera, second depth camera, as taught by Zhao into the system of Goldberg et al., because such incorporation would allow for the benefit of providing more options to a user to have 3D images, thus increase user accessibility of the system.
Regarding claim 2, the method wherein the determining the object coordinate of the object three-dimensional coordinate and the eye coordinate of the eye three-dimensional coordinate under each collection camera coordinate system comprises:
in response to that the first depth camera coordinate system is unified with the second depth camera coordinate system, determining the object coordinate of the object three-dimensional coordinate under each collection camera coordinate system based on an external parameter matrix from the first depth camera coordinate system to each collection camera coordinate system (in addition to discussion above, Goldberg et al., paragraph 0163; Zhao, col. 6 lines 38-50 teaches “Calibrating the first depth cameras 201 can include a process of determining their internal and external parameters. The internal parameters can be inherent to the first depth cameras 201, including for example internal geometrical and optical parameters independent of positional parameters of the first depth cameras 201. For example, the internal parameters can include coordinates of a center of images, a focal length, a scale factor, the lens distortions, etc. The external parameters can include a transformation matrix and a translation matrix used to determine a three-dimensional location and direction relationship between the coordinate system of the first depth camera 201 and a specified world coordinate system.”); and
determining the eye coordinate of the eye three-dimensional coordinate under each collection camera coordinate system based on an external parameter matrix from the second depth camera coordinate system to each collection camera coordinate system (in addition to discussion above, Goldberg et al., paragraph 0163; Zhao, col. 9 lines 11-35 teaches “The second image capturing device can be located over the first image capturing device 20 with a shooting angle directed upward. The second image capturing device can also comprise a plurality of second depth cameras. In some embodiments, the central computing device, taking a plurality of images captured at the same time by all the first depth cameras 201 as one frame, based on the calibration parameters of the first depth cameras 201, performs point cloud stitching. More specifically, this process can include: the central computing device, taking a plurality of images captured at the same time by all the first depth cameras 201 of both the first image capturing device 20 and the second image capturing device (not shown) as one frame, based on the calibration parameters of the first depth cameras 201, performs point cloud stitching. In some embodiments, advanced calibration can be performed for the second image capturing device according to the placement location and the angle of each depth camera 201 of the second image capturing device, as in the case for the first image capturing device 20.”).
The motivation for combining references has been discussed in independent claim above.
Regarding claim 7, the method wherein the determining the line-of-sight direction data of the user based on the object coordinate and the eye coordinate under each collection camera coordinate system comprises:
determining a line-of-sight vector under each collection camera coordinate system based on the object coordinate and the eye coordinate under each collection camera coordinate system (in addition to discussion above, Goldberg et al., paragraph 0153 teaches “estimating a first line-of-sight (LOS) vector in a three-dimensional coordinate system for a first of the user's eyes based on the image data captured by the single image capturing device; estimating a second LOS vector in the three-dimensional coordinate system for a second of the user's eyes based on the image data captured by the single image capturing device; determining the three-dimensional POG of the user in the scene in the three-dimensional coordinate system using the first and second LOS vectors as estimated based on the image data captured by the single image capturing device”; Zhao, col. 5 lines 37-50 teaches “In some embodiments, the first depth cameras 201 can be realized using conventional cameras. In some other embodiments, light-field cameras can be employed to capture both light intensities, color, directional, and/or phase information.… The wide-angle coverage can produce panoramic images or views. … For example, the plurality of first depth cameras 201 can be disposed at edges of the target area. The plurality of first depth cameras 201 of the first image capturing device 20 can capture images of the area from various directions, such as in the inward directions from various positions outside the target area.”); and
determining the line-of-sight direction data of the user based on the line-of-sight vector under each collection camera coordinate system (in addition to discussion above, Goldberg et al., paragraph 0153; Zhao, col. 5 lines 37-50).
The motivation for combining references has been discussed in independent claim above.
Regarding claim 8, a device for collecting line-of-sight direction data, comprising: a memory, a processor, and a line-of-sight direction data collection program stored in the memory and executable on the processor, wherein the line-of-sight direction data collection program is configured to implement steps of the method for collecting line-of-sight direction data according to claim 1 (in addition to discussion above, Goldberg et al., paragraph 0020; Zhao, fig. 9).
Regarding claim 9, the device wherein the determining the object coordinate of the object three-dimensional coordinate and the eye coordinate of the eye three-dimensional coordinate under each collection camera coordinate system comprises:
in response to that the first depth camera coordinate system is unified with the second depth camera coordinate system, determining the object coordinate of the object three-dimensional coordinate under each collection camera coordinate system based on an external parameter matrix from the first depth camera coordinate system to each collection camera coordinate system (in addition to discussion above, Goldberg et al., paragraph 0163; Zhao, col. 9 lines 11-35 teaches “The second image capturing device can be located over the first image capturing device 20 with a shooting angle directed upward. The second image capturing device can also comprise a plurality of second depth cameras. In some embodiments, the central computing device, taking a plurality of images captured at the same time by all the first depth cameras 201 as one frame, based on the calibration parameters of the first depth cameras 201, performs point cloud stitching. More specifically, this process can include: the central computing device, taking a plurality of images captured at the same time by all the first depth cameras 201 of both the first image capturing device 20 and the second image capturing device (not shown) as one frame, based on the calibration parameters of the first depth cameras 201, performs point cloud stitching. In some embodiments, advanced calibration can be performed for the second image capturing device according to the placement location and the angle of each depth camera 201 of the second image capturing device, as in the case for the first image capturing device 20.”).; and
determining the eye coordinate of the eye three-dimensional coordinate under each collection camera coordinate system based on an external parameter matrix from the second depth camera coordinate system to each collection camera coordinate system (in addition to discussion above, Goldberg et al., paragraph 0163; Zhao, col. 9 lines 11-35).
The motivation for combining references has been discussed in independent claim above.
Claim 14 is rejected for the same reason as discussed in the corresponding claim 7 above.
Claim 15 is rejected for the same reason as discussed in the corresponding claim 8 above.
Claim 16 is rejected for the same reason as discussed in the corresponding claim 9 above.
Claim 20 is rejected for the same reason as discussed in the corresponding claim 7 above.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NIGAR CHOWDHURY whose telephone number is (571)272-8890. The examiner can normally be reached Monday-Friday 9AM-5PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Thai Tran can be reached at 571-272-7382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/NIGAR CHOWDHURY/Primary Examiner, Art Unit 2484
Prosecution Insights