Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 02/12/2026 has been entered.
Response to Arguments
Applicant’s arguments with respect to claim(s) 1-4, 6-20 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 4, 8, 11, 14-16, 19 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Agrawal et al. (US PGPUB 20230276115), Naserian et al. (US PGPUB 20230034777) and in further view of Sinha et al. (US PGPUB 20240267494).
[Claim 1]
Agrawal teaches a computer-implemented method, comprising: causing a camera of a device to capture a first video of an environment, wherein the camera is an outwardly-facing camera facing the environment and facing away from a user (Paragraph 90, Referring to FIG. 6C, electronic device 100 is further shown continuing video communication session 386 by user 610. In FIG. 6C, user 610 has looked away from front surface 176 and display 130 in a second eye gaze direction 260B towards dog 626. Dog 626 is located in second location 262B, which is offset to the rear of electronic device 100 at a distance 644 away from electronic device 100. Dog 626 is located facing rear surface 180 and is in a FOV 646 of at least one of rear facing cameras 133 A-C (FIG. 1C)), and wherein the device comprises:
a camera direction control element for controlling a capture direction of the camera (Paragraph 98, In response to determining that the user is looking towards the rear of the device and identifying the corresponding location, AI engine 214 enables electronic device 100 to generate at least one new camera setting 266 of the active camera (i.e., rear camera 133A) based on the current eye gaze direction (i.e., third eye gaze direction 260C) and adjust the active camera (i.e., rear camera 133A) using the generated at least one camera setting 266 and Paragraph 99, In one embodiment, adjusting the active camera using the generated new camera settings 266 can include adjusting a directional setting of the rear facing camera 133A to focus the FOV 662 of rear facing camera 133A on the current location (i.e., third location 262C) determined from the current eye gaze detection (i.e., third eye gaze direction 260C)); and
a camera zoom control element for controlling zoom of the camera (Paragraph 55, zoom lens);
detecting a gaze angle of the user having the computing device (Paragraph 88, CSCM 136 enables electronic device 100 to recognize the face 614 of a user and to track the eyes 616 of the user to determine first eye gaze direction 260A. More specifically, CSCM 136 enables electronic device 100 to detect the presence of eyes within a FOV of front facing camera 132A, to identify precise eye positions and to determine an eye gaze direction based on the eye positions);
identifying, based on the detected gaze angle, one or more objects in the captured first video (Paragraph 92, CSCM 136 further enables processor 102 of electronic device 100 to determine if the user is looking away from the front surface 176 and towards a location behind electronic device 100 (i.e., towards a location that can be captured within a FOV of at least one of the rear facing cameras) for more than a threshold amount of time. In response to processor 102 determining that the user's eye gaze is in a direction within the FOV of at least one of the rear cameras, and in part based on a context determined from processing, by NLP/CE engine, of detected speech of the user, processor 102 triggers ICD controller 134 (FIG. 1A) to select as an active camera, a corresponding one of at least one rear facing cameras 133A-133C with a FOV 646 towards second eye gaze direction 260B and containing second location 262B to which the user 610 is looking. In the example of FIG. 6C, rear facing main camera 133A can be selected as the active camera with a FOV 646 that includes dog 626 when the user says “my dog is so adorable” contemporaneously with fixing his/her gaze in direction of dog 626. Paragraph 67, Image characteristics 272 are attributes identified as associated with an eye gaze location 262 and are used to adjust one or more camera settings to focus the camera on the eye gaze location. For example, image characteristics 272 can include light levels at the eye gaze location, face detection at the eye gaze location and scene detection at the eye gaze location.);
determining, based on the identified one or more objects, a target location in the environment (Paragraph 98, In response to determining that the user is looking towards the rear of the device and identifying the corresponding location, AI engine 214 enables electronic device 100 to generate at least one new camera setting 266 of the active camera (i.e., rear camera 133A) based on the current eye gaze direction (i.e., third eye gaze direction 260C) and adjust the active camera (i.e., rear camera 133A) using the generated at least one camera setting 26);
adjusting the capture direction of the camera using the camera direction control element based on the determined target location in the environment (Paragraph 99, In one embodiment, adjusting the active camera using the generated new camera settings 266 can include adjusting a directional setting of the rear facing camera 133A to focus the FOV 662 of rear facing camera 133A on the current location (i.e., third location 262C) determined from the current eye gaze detection (i.e., third eye gaze direction 260C).; a
adjusting the zoom of the camera using the camera zoom control element based on the determined target location in the environment (Paragraph 99, In another embodiment, adjusting the active camera using the generated new camera settings 266 can include adjusting a zoom level of the rear facing camera 133A to magnify captured images of the current location (i.e., third location 262C)); and
causing the camera to capture a second video using the camera of the device, wherein the second video is captured based on the adjusted capture direction and the adjusted zoom of the camera (Paragraph 101, The captured video image 672 of cat 624 captured by rear facing main camera 133A is shown in captured image/video window 632 on display 130. AI engine 214 enables electronic device 100 to generate optimal camera settings 268 and to adjust the camera settings such that captured video image 672 is optimized for size (zoom) and position (directional setting). The received video stream 634 of a user of second electronic device 300 is shown in received image/video window 636 on display 130).
Agarwal fails to teach determining that the gaze angle indicates that a gaze of the user is directed at different objects of the identified one or more objects over a plurality of frames of the first video, computing a weighted center point in the environment based on the gaze of the
user over the plurality of frames of the first video; and identifying the weighted center point as the target location in the environment.
However Naserian teaches in FIG. 5 depicts an illustration 500 of a vehicle 100 illuminating a road hazard 510 and a VRU 520 according to one or more embodiments described herein. Particularly, FIG. 5 relates to the blocks 416, 418, and 420 of FIG. 4, namely to create a bounding box around a target at block 416, determine a centroid from the bounding box based on weight of the object and/or the driver gaze at block 418, and generate a light pattern to make the centroid area salient at block 420 (Paragraph 60). In this example, the vehicle 100 is shown traveling along a path/trajectory 502. An object map 504 is also shown, bounded about an area surrounding and extending around the vehicle 100. In this example, two objects 510, 520 are detected. The road hazard 510 represents a road hazard in the form of another vehicle. The object 520 represents a VRU (Paragraph 61). Bounding boxes 512, 522 are created around the objects 510, 520 respectively. According to one or more embodiments described herein, the bounding boxes 512, 522 are sized such that they are approximately the same size of the respective objects 510, 520. A center point (or “centroid”) for each of the bounding boxes 512, 522 is generated. For example, the bounding box 512 has a center point 514, and the bounding box 522 has a center point 524. According to one or more embodiments described herein, the center points 514, 524 are unweighted such that they are in the center of the respective bounding boxes 512, 522 (Paragraph 62). A grouped bounding box 532 can be created based on the bounding boxes 512, 522. According to one or more embodiments described herein, the grouped bounding box 532 encompasses each of the bounding boxes 512, 522. A center point 534 (or “centroid”) of the grouped bounding box 532 can be determined. According to one or more embodiments described herein, the center point 534 is not in the center of the bounding box 532; rather, it is biased based on the importance of the objects 510, 520. In this example, the center point 534 is biased towards the VRU (i.e., the object 520) as shown. According to one or more embodiments described herein, the weighting for the type of objects can be based on a type of the object (e.g., VRU vs. road hazard) and a risk associated with each object. For example, a pothole with a large magnitude may have a higher weight than a pothole with a small magnitude. Similarly, a VRU may have a higher weight than a non-VRU, such as shown in FIG. 5 (Paragraph 63). In this example, headlights (not shown) of the vehicle 100 can be configured to make a weighted mean center (e.g., the center point 534) of the grouped bounding box 532 more prominent. In some examples, the center point 534 can also be based on a projected gaze of a driver of the vehicle 100 (Paragraph 64).
Therefore taking the combined teachings of Agrawal and Naserian, it would be obvious to one skilled in the art before the effective filing date of the invention to have been motivated to have determining that the gaze angle indicates that a gaze of the user is directed at different objects of the identified one or more objects over a plurality of frames of the first video, computing a weighted center point in the environment based on the gaze of the
user over the plurality of frames of the first video; and identifying the weighted center point as the target location in the environment in order to illuminate the most vulnerable or desirable object so that they can be photographed and protected.
Agrawal in view of Naserian fails to teach a head-mounted computing device.
However Sinha teaches headset 2 (or head-mounted device (HMD)) is eye glasses that are being worn by the user 1. In one embodiment, the headset may be any type of device that may be worn on a user's head, such as an eyepatch, goggles, a visor, headgear, headphones, or otherwise. In another embodiment, the headset may be any type of electronic device, which may or may not be a part of (or worn on) the user's head (Paragraph 35) and the headset 2 includes a camera 6, an antenna mount 5 to which one or more antennas 10 are coupled, and an input device 11 (which may include a user interface). In particular, each or least some of the elements of the headset may be coupled to the frame 4 of the headset. In this example, the frame is a glasses frame, where the elements are a part of (or integrated into) the frame. In one embodiment, the headset may include more or less elements, such as having more or less cameras (Paragraph 36).
Therefore taking the combined teachings of Agrawal, Naserian and Sinha, it would be obvious to one skilled in the art before the effective filing date of the invention to have been motivated to have a headset as the user device in order to have hands free while taking images thereby leading to more enjoyable experience.
[Claim 4]
Agrawal teaches wherein adjusting the capture direction of the camera using the camera direction control element is performed without receiving a direct user request to modify the camera capture direction (Paragraph 94, According to one aspect of the disclosure, electronic device 100 can automatically monitor an eye gaze direction and select which specific rear camera to use when the eye gaze direction of a user changes to looking away from the front surface of electronic device 100 in a direction that is towards the rear of the device and/or within a FOV of one of the rear cameras. By automatically selecting which rear camera to use, electronic device 100 frees the user from the task of having to manually switch to a rear camera when the user's eye gaze location changes or when communication context calls for such selection) and adjusting the zoom of the camera using the camera zoom control element is performed without receiving a direct user request to modify the zoom of the camera and Paragraph 102, According to one additional aspect of the disclosure, electronic device 100 can be used to automatically track the eye gaze direction of a user by monitoring and periodically analyzing image 246 and to adjust camera settings when the current eye gaze direction of a user changes to a different eye gaze direction. By automatically tracking the eye gaze direction of the user, electronic device 100 frees the user from the task of having to manually zoom and pan images using touch screen 131 during video communication session 386 or during a video recording).
[Claim 8]
Agrawal teaches wherein the capture direction of the camera is initially set to correspond to the detected gaze angle; and the zoom of the camera is initially set to a predefined zoom level (Paragraph 102).
[Claim 11]
Agrawal teaches wherein adjusting the capture direction of the camera further comprises: determining an intersection point of respective viewing directions of the eyes of the user (Paragraph 88, According to one aspect of the disclosure, CSCM 136 enables electronic device 100 to recognize the face 614 of a user and to track the eyes 616 of the user to determine first eye gaze direction 260A. More specifically, CSCM 136 enables electronic device 100 to detect the presence of eyes within a FOV of front facing camera 132A, to identify precise eye positions and to determine an eye gaze direction based on the eye positions); and computing the adjusted capture direction based at least in part on the intersection point (Paragraph 102, According to one additional aspect of the disclosure, electronic device 100 can be used to automatically track the eye gaze direction of a user by monitoring and periodically analyzing image 246 and to adjust camera settings when the current eye gaze direction of a user changes to a different eye gaze direction. By automatically tracking the eye gaze direction of the user, electronic device 100 frees the user from the task of having to manually zoom and pan images using touch screen 131 during video communication session 386 or during a video recording.).
[Claim 14]
Sinha teaches causing at least one of the first video or the second video to be captured in response to detecting a particular blink pattern of an eye of the user (For instance, the controller may determine whether to zoom based on detecting a user's gaze and detecting one or more eye movements. In another embodiment, the detected gesture may be any type of physical gesture, such as a hand movement, a head movement, or a facial expression. Upon detecting the gesture, the controller may proceed to determining the region of interest at decision block 115, Paragraph 127) in order to have a different command for capturing videos thereby giving more versatility in the operation of the device.
[Claim 15]
Agrawal teaches determining that the first video depicts a particular type of subject matter, wherein each of adjusting the capture direction, and adjusting the zoom of the camera, is performed based at least in part on determining that the first video depicts the particular type of subject matter (Paragraph 139, Processor 102 retrieves the current eye gaze direction (i.e., eye gaze direction 260) and the current eye gaze location (i.e., location 262) from system memory 120 (block 1220). Processor 102 selects, based on the current eye gaze direction, the current location, and the identified context, a corresponding one of the rear facing cameras 133A-133C as an active camera (block 1222). For example, if the first direction eye gaze direction 260A is facing in the direction of rear surface 180 of electronic device 100 and first location 262A is 5 degrees left and 15 degrees down, with an audio communication input 222 of “clothing” or “dress”, then one of the rear facing cameras 133A-133C with FOV of the “clothing” or “dress” at that location can be selected as the active camera.).
[Claims 16, 19 and 20]
These are apparatus claims corresponding to method claims 1, 4 and 5 and are therefore analyzed and rejected based upon method claims 1, 4 and 5.
Claim(s) 2 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Agrawal et al. (US PGPUB 20230276115), Naserian et al. (US PGPUB 20230034777), Sinha et al. (US PGPUB 20240267494) and in further view of Fiske (US PGPUB 20170242225).
[Claim 2]
Agrawal, Naserian in view of Sinha fails to teach wherein the camera direction control element comprises a microelectromechanical systems (MEMS) scanning mirror, and adjusting the capture direction of the camera using the camera direction control element comprises modifying an orientation of the MEMS scanning mirror. However Fiske teaches a MEMS scan mirror 120 in FIG. 21 depicts the image scanning procedure used by camera module 170. With mirror segment 120 rotated to the designated scan limit in one direction, image sensor 165 captures a first image segment and transfers the first image segment to control processor 202. Control processor 202 then provides the appropriate current direction and amplitude through mirror segment 120 to rotate the mirror segment one third of the way toward the second designated scan limit, where image sensor 165 captures a second image segment and transfers the second image segment to control processor 202. This process repeats for image segments three and four, after which mirror segment 120 has reached the second designated scan limit from where mirror segment 120 started. Mirror rotation angle is precisely controlled and image segment capture is precisely timed such that each successive image segment overlaps the previous image segment to a small degree. In this first embodiment, scan mirror 120 pauses at four positions for as much time as is needed for image sensor 165 to capture an image segment (Paragraph 73).
Therefore taking the combined teachings of Agrawal, Naserian, Sinha and Fiske, it would be obvious to one skilled in the art before the effective filing date of the invention to have been motivated to have wherein the camera direction control element comprises a microelectromechanical systems (MEMS) scanning mirror, and adjusting the capture direction of the camera using the camera direction control element comprises modifying an orientation of the MEMS scanning mirror in order to assist with optical image stabilization and/or improved resolution in response to orientation information, mirror segment may be rotated by small angles to compensate for camera shake around one axis.
[Claim 17]
This is an apparatus claims corresponding to method claim 2 and is therefore analyzed and rejected based upon method claim 2.
Claim(s) 3 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Agrawal et al. (US PGPUB 20230276115), Naserian et al. (US PGPUB 20230034777), Sinha et al. (US PGPUB 20240267494) and in further view of Chen et al. (US PGPUB 20240142831).
[Claim 3]
Agrawal in view of Sinha fails to teach wherein the camera zoom control element comprises a liquid lens, and adjusting the zoom of the camera using the camera zoom control element comprises applying an electrical signal to the liquid lens. However Chen teaches that by adjusting the values of the two voltages, continuous adjustment of the arrangement direction of the liquid crystal molecules can be achieved, i.e., continuous adjustments of a positive lens and a negative lens are achieved, and the zooming of the camera is achieved accordingly. Thus, the liquid crystal molecules in the first liquid crystal layer 310 can be deflected at different degrees, thereby achieving the refraction of the light and the lens zooming function of the camera (Paragraph 38). Therefore taking the combined teachings of Agrawal, Naserian, Sinha and Fiske, it would be obvious to one skilled in the art before the effective filing date of the invention to have been motivated to have the camera zoom control element comprises a liquid lens, and adjusting the zoom of the camera using the camera zoom control element comprises applying an electrical signal to the liquid lens in order to improve the integration level of the device, and facilitate the use and the manufacturing of the device.
[Claim 18]
This is an apparatus claims corresponding to method claim 3 and is therefore analyzed and rejected based upon method claim 3.
Claim(s) 9 is rejected under 35 U.S.C. 103 as being unpatentable over Agrawal et al. (US PGPUB 20230276115), Naserian et al. (US PGPUB 20230034777), Sinha et al. (US PGPUB 20240267494) and in further view of Yoshida et al. (US PGPUB 20230007232).
[Claim 9]
Agrawal, Naserian in view of Sinha fails to teach inputting, to a trained machine learning model, data comprising one or more detected gaze angles of the user over a plurality of frames of the first video and images corresponding to the plurality of frames of the first video; and receiving as output from the trained machine learning model, based on the input to the trained machine learning model, a desired zoom of the camera and a desired capture direction of the camera, wherein adjusting the zoom of the camera is performed based on the desired zoom of the camera, and adjusting the capture direction of the camera is performed based on the desired capture direction of the camera. However Yoshida teaches in a case where the framing and zooming processing based on the object gazed by the user is performed in the video signal processing unit 105, a position, a size, and a direction in which each object is displayed in the video frame change from those in an original video frame (Paragraph 124). The video signal processing unit 105 is assumed to perform the video signal processing as described above by the machine learning model. It is expected to implement optimal video signal processing by using the machine learning model preliminary learning of which by deep learning is performed by the artificial intelligence server on the cloud (Paragraph 76). Therefore taking the combined teachings of Agrawal, Naserian, Sinha and Yoshida, it would be obvious to one skilled in the art before the effective filing date of the invention to have been motivated to have a trained machine learning model, data comprising one or more detected gaze angles of the user over a plurality of frames of the first video and images corresponding to the plurality of frames of the first video; and receiving as output from the trained machine learning model, based on the input to the trained machine learning model, a desired zoom of the camera and a desired capture direction of the camera, wherein adjusting the zoom of the camera is performed based on the desired zoom of the camera, and adjusting the capture direction of the camera is performed based on the desired capture direction of the camera in order to implement a function of artificially implementing, by software or hardware, a function exhibited by a human brain such as learning, inference, data creation, and planning. The function of the artificial intelligence may be implemented using a machine learning model represented by a neural network that simulates a human cranial nerve circuit.
Claim(s) 10 is rejected under 35 U.S.C. 103 as being unpatentable over Agrawal et al. (US PGPUB 20230276115), Naserian et al. (US PGPUB 20230034777), Sinha et al. (US PGPUB 20240267494) and in further view of Wong et al. (US Patent # 9,949,637).
[Claim 10]
Agrawal, Naserian in view of Sinha fails to teach wherein the head-mounted computing device further comprises a beam splitter, the method further comprising: using the beam splitter to cause an optical center of the camera to correspond to a position of an eye of the user, to enable determining the adjusted capture direction based on the detected gaze angle. However Wong teaches the gaze direction 204 passes through a proximal beam splitter 222. The eye 200 of the wearer may be located on one side of proximal beam splitter 222, and the other side of the proximal beam splitter 222 may be provided with a viewing window 224 that allows light into the proximal beam splitter 222 from outside of the optical system 210. In this way, the wearer is able to view the real world through the viewing window 224 and the proximal beam splitter 222, along the gaze direction 204 (col. 6 lines 35-44). Therefore taking the combined teachings of Agrawal, Naserian, Sinha and Wong, it would be obvious to one skilled in the art before the effective filing date of the invention to have been motivated to have a beam splitter, the method further comprising: using the beam splitter to cause an optical center of the camera to correspond to a position of an eye of the user in order to determine the direction, angle, and/or other properties of the change in the gaze direction of the eye using the images before and after the change in the gaze direction of the eye. Further, images of the eye could be taken at more than two points in time, and the images could be used to detect a change in the gaze direction of the eye and the determine properties of the change in the gaze direction of the eye.
Allowable Subject Matter
Claim 6 is allowed. Reasons for allowance were cited in a previous office action.
Claims 7, 12 and 13 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims. Reasons for allowance were cited in a previous office action.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to YOGESH K AGGARWAL whose telephone number is (571)272-7360. The examiner can normally be reached Monday - Friday 9:30-6.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sinh Tran can be reached at 5712727564. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/YOGESH K AGGARWAL/Primary Examiner, Art Unit 2637