DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The preliminary amendment to the claims filed on 2/10/2026 has been entered. The claims 1 and 10 have been amended. The claims 1-15 are pending in the current application.
Response to Arguments
Applicant's arguments filed 2/10/2026 have been fully considered but they are not persuasive.
In Remarks, applicant repeated the claim language of “maintaining the pose of the at least one camera at the time of capturing the real-world image with the representation” and argued with a specific embodiment in the prior art references. However, the examiner does not reply upon the change of the pose embodiment in the prior art reference.
In other words, Pizzocchero teaches the new claim limitation of maintaining the pose of the at least one camera at the time of capturing the real-world image with the representation.
In a non-limiting example, at the time of capturing the real-world image, the capture setting such as the pose of the camera of Pizzocchero is maintained. The capture setting is not associated with the pose of the image capture device 102 without rotation/tilt of the image capture device 102 for a single frame. The capture setting is the flash intensity, flash duration, shutter speed, ISO speed, gain, aperture, light balance, other than the pose of the image capture device 102.
Pizzocchero teaches at Paragraph 0052 that, the physical document and mobile computing device 102 remains stationary during video capture but one or more operational parameters and/or capture settings of image capture device 103 (e.g., flash intensity, flash duration, shutter speed, ISO speed, gain, aperture, light balance, etc.) are modified or adjusted for different frames of the video, in order to cause one or more security features (such as OVDs) on the physical document to become visible or invisible, change color, change appearance, and so forth. In some embodiments, mobile computing device 102 analyzes frames of the video as the frames are being captured and automatically adjusts operational parameters and/or capture settings of image capture device 103 to generate a set of frames with varying imaging conditions, lighting conditions, and/or image characteristics.
Rao teaches the new claim limitation of maintaining the pose of the at least one camera at the time of capturing the real-world image with the representation.
In a non-limiting example, at the time of capturing the video, the pose of the video camera is maintained and camera parameters are not associated with the pose of the image capture device 102 without rotation/tilt of the video camera for a single frame. The capture setting is the brightness, color, contrast and sharpness (see Rao Paragraph 0024) while the camera system 202 of FIG. 2 is in the fixed position (fixed pose).
Rao teaches at Paragraph 0097 that camera parameters (e.g., brightness, color, contrast, and sharpness), can be dynamically and automatically adjusted based on the reward function derived from the perceptual quality estimator.
The camera parameter of Rao is associated with brightness, color, contrast and sharpness and is not associated with the pose of the camera.
Rao teaches at Paragraph 0028 that the adaptive perceptual camera tuning (APT) system can utilize a neural network and reinforcement learning (RL) to automatically and adaptively tune camera parameters remotely to generate a high-quality video feed and at Paragraph 0096 that a video/video stream can be acquired using one or more video cameras deployed in various environmental settings in block 502 for use (e.g., real-time use) in video analytics applications (e.g., object/person detection).
Rao teaches Page 13, column 2, lines 13-16 that the camera parameter settings are tuned responsive to a perceptual no-reference quality determination indicating a quality of the captured frames being below a quality threshold level.
Grant teaches the new claim limitation of maintaining the pose of the at least one camera at the time of capturing the real-world image with the representation.
In other words, increasing resolution or increasing focus in Grant involves changing the zoom level or changing the focal length (while maintaining the pose of the image capture device) and does not involve changing the image capture parameter such as the pose of the image capture device.
Grant teaches at Paragraph [0049] the sensor data pertains to environmental factors within a predefined vicinity of image capture, e.g., weather or air quality, based upon which the image analysis application may facilitate further adjustment. For instance, the image analysis application may adjust contrast of the at least one captured image and/or the plurality of discrete objects therein upon determining via sensor data that the weather is cloudy. In another instance, the image analysis application may adjust color of the at least one captured image and/or the plurality of discrete objects therein upon determining via sensor data that the air quality is low due to smoke.
Grant teaches at Paragraph [0066] that he emphasis of one discrete object relative to the others may entail emphasizing one or more photographic parameters of such discrete object (e.g., increasing resolution, increasing focus, increasing contrast, etc.). Per step 230, the image analysis application presents to Client A the captured image modification options, e.g., via a viewfinder of the image capture device. Accordingly, Client A may select one or more of the captured image modification options, depending upon whether Client A prefers emphasis of the oak tree, emphasis of the mountain, or (optionally) emphasis of both the oak tree and the mountain in the at least one captured image.
Grant teaches at Paragraph 0050 that the image analysis application may facilitate adjustment of the pre-capture image parameters of the image capture device such that the image capture device focuses upon, zooms in upon, and/or otherwise emphasizes the tiger for purposes of subsequent image capture. The image analysis application optionally facilitates adjustment of at least one pre-capture image parameter in view of the image capture learning model prior to capture of a subsequent image based upon data collected by the at least one monitoring sensor. For instance, assuming that the client discusses a deer while sitting in a car and aims the image capture device toward the deer, based upon sensor data collected pertaining to the client deer discussion, the image analysis application optionally facilitates adjustment of at least one pre-capture image parameter such that the image capture device focuses upon, zooms in upon, and/or otherwise emphasizes the deer for purposes of subsequent image capture.
Grant teaches at Paragraph 0058 that the image analysis application determines resolution adjustment patterns among respective discrete objects in one or more previously captured images by applying at least one classification machine learning algorithm and/or at least one clustering machine learning algorithm. The image analysis application determines type and/or degree of resolution adjustment (e.g., type and/or degree of a blurring effect) among respective discrete objects by applying a classification algorithm. Additionally or alternatively, the image analysis application determines level of resolution adjustment among respective discrete objects by applying a quality threshold (QT) clustering algorithm.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-15 are rejected under 35 U.S.C. 103 as being unpatentable over Agrawal et al. US-PGPUB No. 2023/0298197 (hereinafter Agrawal) in view of Pizzocchero et al. US-PGPUB No. 2023/0281821 (hereinafter Pizzocchero);
Rao et al. US-PGPUB No. 2024/0089592 (hereinafter Rao);
Malin et al. US-PGPUB No. 2025/0095204 (hereinafter Malin);
and Grant et al. US-PGPUB No. 2021/0211575 (hereinafter Grant).
Re Claim 1:
Agrawal teaches a method comprising:
determining a gaze point and a gaze depth of a user's eyes, by processing gaze-tracking data that is collected by a gaze-tracking means (
Agrawal teaches at Paragraph 0031 that CCM 136 includes program code that is executed by processor 102 to enable electronic device 100 to adjust camera settings based on eye gaze tracking and at Paragraph 0049 that CCM 136 enables electronic device 100 to track an eye gaze direction of a user within a display and to automatically adjust an active camera based on the eye gaze tracking.
Agrawal teaches at Paragraph 0055 that eye gaze location 262 is a specific location that a user of electronic device 100 is looking, which correlates to and is determined based on the eye gaze direction 260. Eye gaze location 262 can include a plurality of discrete locations, such as first location 262A, second location 262B, and third location 262C. According to one embodiment, eye gaze location 262 can include the orientation to the location where the user is looking based on the corresponding eye gaze direction. For example, eye gaze location 262 can have a vertical orientation of 5 degrees upwards, and a horizontal direction that is positive (right) 15 degrees from the front surface 176 of electronic device 100. It is appreciated that the dimensions and orientations measurements would be based on the front surface of electronic device 100 representing a vertical/Y plane, in a 3-dimensional X-Y-Z axes coordinate plane extending through a center point of the front surface of electronic device 100.
Agrawal teaches at Paragraph [0069] that, referring to FIG. 3E, electronic device 100 is further shown after electronic device 100 has (i) determined, based on detection of second eye gaze direction 260B towards second location 262B within display 130, that there is a new ROI, (ii) generated at least one new camera setting and (iii) adjusted at least one camera setting of the active rear facing camera. Specifically, CCM 136 enables electronic device 100 to map second location 262B (i.e., grid 139S) on display 130 to a new second ROI 250B containing dog 326. CCM 136 further enables electronic device 100 to generate at least one new camera setting 266 of active rear facing camera 133A at least partially based on the new second ROI 250B. CCM 136 further enables electronic device 100 to adjust the active rear facing camera 133A using the generated new camera setting 266 such that the rear facing camera 133A focuses on the new second ROI 250B. In one embodiment, the generated camera settings 266 can be optimal camera settings 268 that provide an optimal sized and zoomed image on display 130 of the video recording of second ROI 250B.
Agrawal teaches at Paragraph [0070] In one embodiment, adjusting the active rear facing camera using the generated new camera settings 266 can include adjusting a directional setting (pose) of the rear facing camera 133A to focus the FOV 362 of rear facing camera 133A on the new second ROI 250B that is determined from the current eye gaze location (i.e., second location 262B). In another embodiment, adjusting the active rear facing camera using the generated new camera settings 266 can include adjusting a zoom level of the rear facing camera 133A to magnify or reduce captured images of the new ROI (i.e., second ROI 250B). In an additional embodiment, adjusting the active rear facing camera using the generated new camera settings 266 can include adjusting a focal distance of the rear facing camera 133A to focus the rear facing camera on the new ROI (i.e., second ROI 250B). In one embodiment, a different one of the rear-facing cameras can be selected to be the active rear camera in instances in which the previous active rear camera is unable to provide a good or complete view of the ROI within the display. For example, when the ROI is at a periphery of the FOV of a standard rear camera, a wide angled camera can be selected to be the active rear camera to be able to focus on the entire ROI without having to crop the edges of the video images. Other camera settings adjustments can be made based on the detected amount of light, exposure settings and other identified image characteristics of the new ROI);
controlling at least one camera for capturing a real-world image of a real-world environment, by adjusting camera settings according to the gaze point and the gaze depth (
Agrawal teaches at Paragraph 0031 that CCM 136 includes program code that is executed by processor 102 to enable electronic device 100 to adjust camera settings based on eye gaze tracking and at Paragraph 0049 that CCM 136 enables electronic device 100 to track an eye gaze direction of a user within a display and to automatically adjust an active camera based on the eye gaze tracking.
Agrawal teaches at Paragraph 0055 that eye gaze location 262 is a specific location that a user of electronic device 100 is looking, which correlates to and is determined based on the eye gaze direction 260. Eye gaze location 262 can include a plurality of discrete locations, such as first location 262A, second location 262B, and third location 262C. According to one embodiment, eye gaze location 262 can include the orientation to the location where the user is looking based on the corresponding eye gaze direction. For example, eye gaze location 262 can have a vertical orientation of 5 degrees upwards, and a horizontal direction that is positive (right) 15 degrees from the front surface 176 of electronic device 100. It is appreciated that the dimensions and orientations measurements would be based on the front surface of electronic device 100 representing a vertical/Y plane, in a 3-dimensional X-Y-Z axes coordinate plane extending through a center point of the front surface of electronic device 100.
Agrawal teaches at Paragraph [0069] that, referring to FIG. 3E, electronic device 100 is further shown after electronic device 100 has (i) determined, based on detection of second eye gaze direction 260B towards second location 262B within display 130, that there is a new ROI, (ii) generated at least one new camera setting and (iii) adjusted at least one camera setting of the active rear facing camera. Specifically, CCM 136 enables electronic device 100 to map second location 262B (i.e., grid 139S) on display 130 to a new second ROI 250B containing dog 326. CCM 136 further enables electronic device 100 to generate at least one new camera setting 266 of active rear facing camera 133A at least partially based on the new second ROI 250B. CCM 136 further enables electronic device 100 to adjust the active rear facing camera 133A using the generated new camera setting 266 such that the rear facing camera 133A focuses on the new second ROI 250B. In one embodiment, the generated camera settings 266 can be optimal camera settings 268 that provide an optimal sized and zoomed image on display 130 of the video recording of second ROI 250B.
Agrawal teaches at Paragraph [0070] In one embodiment, adjusting the active rear facing camera using the generated new camera settings 266 can include adjusting a directional setting (pose) of the rear facing camera 133A to focus the FOV 362 of rear facing camera 133A on the new second ROI 250B that is determined from the current eye gaze location (i.e., second location 262B). In another embodiment, adjusting the active rear facing camera using the generated new camera settings 266 can include adjusting a zoom level of the rear facing camera 133A to magnify or reduce captured images of the new ROI (i.e., second ROI 250B). In an additional embodiment, adjusting the active rear facing camera using the generated new camera settings 266 can include adjusting a focal distance of the rear facing camera 133A to focus the rear facing camera on the new ROI (i.e., second ROI 250B). In one embodiment, a different one of the rear-facing cameras can be selected to be the active rear camera in instances in which the previous active rear camera is unable to provide a good or complete view of the ROI within the display. For example, when the ROI is at a periphery of the FOV of a standard rear camera, a wide angled camera can be selected to be the active rear camera to be able to focus on the entire ROI without having to crop the edges of the video images. Other camera settings adjustments can be made based on the detected amount of light, exposure settings and other identified image characteristics of the new ROI);
determining a pose of the at least one camera at a time of capturing the real-world image, by processing pose-tracking data that is collected by a pose-tracking means (
Agrawal teaches at Paragraph 0031 that CCM 136 includes program code that is executed by processor 102 to enable electronic device 100 to adjust camera settings based on eye gaze tracking and at Paragraph 0049 that CCM 136 enables electronic device 100 to track an eye gaze direction of a user within a display and to automatically adjust an active camera based on the eye gaze tracking.
Agrawal teaches at Paragraph 0055 that eye gaze location 262 is a specific location that a user of electronic device 100 is looking, which correlates to and is determined based on the eye gaze direction 260. Eye gaze location 262 can include a plurality of discrete locations, such as first location 262A, second location 262B, and third location 262C. According to one embodiment, eye gaze location 262 can include the orientation to the location where the user is looking based on the corresponding eye gaze direction. For example, eye gaze location 262 can have a vertical orientation of 5 degrees upwards, and a horizontal direction that is positive (right) 15 degrees from the front surface 176 of electronic device 100. It is appreciated that the dimensions and orientations measurements would be based on the front surface of electronic device 100 representing a vertical/Y plane, in a 3-dimensional X-Y-Z axes coordinate plane extending through a center point of the front surface of electronic device 100.
Agrawal teaches at Paragraph [0069] that, referring to FIG. 3E, electronic device 100 is further shown after electronic device 100 has (i) determined, based on detection of second eye gaze direction 260B towards second location 262B within display 130, that there is a new ROI, (ii) generated at least one new camera setting and (iii) adjusted at least one camera setting of the active rear facing camera. Specifically, CCM 136 enables electronic device 100 to map second location 262B (i.e., grid 139S) on display 130 to a new second ROI 250B containing dog 326. CCM 136 further enables electronic device 100 to generate at least one new camera setting 266 of active rear facing camera 133A at least partially based on the new second ROI 250B. CCM 136 further enables electronic device 100 to adjust the active rear facing camera 133A using the generated new camera setting 266 such that the rear facing camera 133A focuses on the new second ROI 250B. In one embodiment, the generated camera settings 266 can be optimal camera settings 268 that provide an optimal sized and zoomed image on display 130 of the video recording of second ROI 250B.
Agrawal teaches at Paragraph [0070] In one embodiment, adjusting the active rear facing camera using the generated new camera settings 266 can include adjusting a directional setting (pose) of the rear facing camera 133A to focus the FOV 362 of rear facing camera 133A on the new second ROI 250B that is determined from the current eye gaze location (i.e., second location 262B). In another embodiment, adjusting the active rear facing camera using the generated new camera settings 266 can include adjusting a zoom level of the rear facing camera 133A to magnify or reduce captured images of the new ROI (i.e., second ROI 250B). In an additional embodiment, adjusting the active rear facing camera using the generated new camera settings 266 can include adjusting a focal distance of the rear facing camera 133A to focus the rear facing camera on the new ROI (i.e., second ROI 250B). In one embodiment, a different one of the rear-facing cameras can be selected to be the active rear camera in instances in which the previous active rear camera is unable to provide a good or complete view of the ROI within the display. For example, when the ROI is at a periphery of the FOV of a standard rear camera, a wide angled camera can be selected to be the active rear camera to be able to focus on the entire ROI without having to crop the edges of the video images. Other camera settings adjustments can be made based on the detected amount of light, exposure settings and other identified image characteristics of the new ROI);
identifying at least one region of the real-world environment that is represented in the real- world image, based on a spatial geometry of the real-world environment and the pose of the at least one camera (
Agrawal teaches at FIGS. 3A, 3C and 3E that the first ROI 250A, the second ROI 250B and the third ROI 250C are configured with different geometric shapes with change of the edges of the ROI and at Paragraph 0061 that the image characteristics of the ROI are identified wherein the image characteristics include face (geometry) detection and scene (geometry) detection enabling focus on the subject (geometry shape) based on the contrast change of the edges in the ROI. The edges of the ROI represent spatial geometry of the ROI as shown in FIGS. 3A-3E with the different geometry shapes outlining the ROIs.
Agrawal teaches at Paragraph [0061] that image characteristics 272 are attributes identified as associated with an ROI 250 and are used to adjust one or more camera settings to focus the camera on the ROI. Image characteristics 272 can be determined from one or more captured ROI images 254 that have been captured by a rear facing camera. For example, image characteristics 272 can include light levels at the ROI, face detection at the ROI, and scene detection at the ROI. Image characteristics 272 can further include contrast detection autofocus that achieves focus on a subject based on the contrast change of the edges in the ROI. Alternatively, image characteristics 272 can include phase detection autofocus that enables focusing on a subject based on the convergence of two separate beams of light. Image characteristics 272 can further include computer vision (CV), machine learning (ML), and artificial intelligence (AI) based techniques for determining an object of interest within the ROI.
Agrawal teaches at Paragraph [0062] FIG. 3A illustrates electronic device 100 being used by a user 310 to record or capture video data 270. A cat 324 and a dog 326 are located behind rear surface 180 away from electronic device 100. The face 314 of the user 310 includes a pair of eyes 316 that are looking in first eye gaze direction 260A. In FIG. 3A, the user is looking at cat 324 and dog 326 along first eye gaze direction 260A towards first ROI 250A. First ROI 250A includes cat 324 and dog 326. Cat 324 and dog 326 are within FOV 346 of the active camera (i.e., rear facing main camera 133A).
Agrawal teaches at Paragraph 0031 that CCM 136 includes program code that is executed by processor 102 to enable electronic device 100 to adjust camera settings based on eye gaze tracking and at Paragraph 0049 that CCM 136 enables electronic device 100 to track an eye gaze direction of a user within a display and to automatically adjust an active camera based on the eye gaze tracking.
Agrawal teaches at Paragraph 0055 that eye gaze location 262 is a specific location that a user of electronic device 100 is looking, which correlates to and is determined based on the eye gaze direction 260. Eye gaze location 262 can include a plurality of discrete locations, such as first location 262A, second location 262B, and third location 262C. According to one embodiment, eye gaze location 262 can include the orientation to the location where the user is looking based on the corresponding eye gaze direction. For example, eye gaze location 262 can have a vertical orientation of 5 degrees upwards, and a horizontal direction that is positive (right) 15 degrees from the front surface 176 of electronic device 100. It is appreciated that the dimensions and orientations measurements would be based on the front surface of electronic device 100 representing a vertical/Y plane, in a 3-dimensional X-Y-Z axes coordinate plane extending through a center point of the front surface of electronic device 100.
Agrawal teaches at Paragraph [0069] that, referring to FIG. 3E, electronic device 100 is further shown after electronic device 100 has (i) determined, based on detection of second eye gaze direction 260B towards second location 262B within display 130, that there is a new ROI, (ii) generated at least one new camera setting and (iii) adjusted at least one camera setting of the active rear facing camera. Specifically, CCM 136 enables electronic device 100 to map second location 262B (i.e., grid 139S) on display 130 to a new second ROI 250B containing dog 326. CCM 136 further enables electronic device 100 to generate at least one new camera setting 266 of active rear facing camera 133A at least partially based on the new second ROI 250B. CCM 136 further enables electronic device 100 to adjust the active rear facing camera 133A using the generated new camera setting 266 such that the rear facing camera 133A focuses on the new second ROI 250B. In one embodiment, the generated camera settings 266 can be optimal camera settings 268 that provide an optimal sized and zoomed image on display 130 of the video recording of second ROI 250B.
Agrawal teaches at Paragraph [0070] In one embodiment, adjusting the active rear facing camera using the generated new camera settings 266 can include adjusting a directional setting (pose) of the rear facing camera 133A to focus the FOV 362 of rear facing camera 133A on the new second ROI 250B that is determined from the current eye gaze location (i.e., second location 262B). In another embodiment, adjusting the active rear facing camera using the generated new camera settings 266 can include adjusting a zoom level of the rear facing camera 133A to magnify or reduce captured images of the new ROI (i.e., second ROI 250B). In an additional embodiment, adjusting the active rear facing camera using the generated new camera settings 266 can include adjusting a focal distance of the rear facing camera 133A to focus the rear facing camera on the new ROI (i.e., second ROI 250B). In one embodiment, a different one of the rear-facing cameras can be selected to be the active rear camera in instances in which the previous active rear camera is unable to provide a good or complete view of the ROI within the display. For example, when the ROI is at a periphery of the FOV of a standard rear camera, a wide angled camera can be selected to be the active rear camera to be able to focus on the entire ROI without having to crop the edges of the video images. Other camera settings adjustments can be made based on the detected amount of light, exposure settings and other identified image characteristics of the new ROI).
Agrawal does not explicitly teach the claim limitation:
determining whether a representation of the at least one region in at least one of: the real- world image, a previously-captured real-world image, satisfies a quality criteria, wherein the previously-captured image is stored at a data repository;
when it is determined that the representation of the at least one region in the at least one of: the real-world image, the previously-captured real-world image, fails to satisfy the quality criteria, controlling the at least one camera for capturing a reference real-world image representing the at least one region, by adjusting the camera settings such that said representation fulfills the quality criteria, while maintaining the pose of the at least one camera at the time of capturing the real-world image with the representation;
generating training data comprising reference data and input data, wherein the reference data comprises the reference real-world image representing the at least one region, and the input data comprises the at least one of: the real-world image of the at least one region, the previously-captured real-world image of the at least one region; and
sending the training data to a processor that is configured to train a first neural network for generating real-world images that satisfy the quality criteria by processing real-world images that fail to satisfy the quality criteria.
However, Malin teaches the new claim limitation: maintaining the pose of the at least one camera at the time of capturing the real-world image with the representation (
Malin teaches at Paragraph 0020 and Paragraph 0041 that the calibration parameters include the focal length, decentering of the optical elements, distortion parameters other than the rotation and displacement parameters for changing the quality of the image (see Paragraph 0043.
It would have been obvious to one of the ordinary skill in the art before the filing date of the instant application to have incorporated Malin’s improvement of the image quality using the camera parameters other than the pose of the camera to achieve quality improvement to have adjusted capture parameters for subsequent frames to achieve image capture that falls within acceptable quality criteria. One of the ordinary skill in the art would have been motivated to have provided camera control module for controlling the camera setting to have focused on the region of interest and for controlling the camera setting to have achieved image capture that falls within acceptable quality threshold.
Pizzocchero explicitly teaches the claim limitation:
determining whether a representation of the at least one region in at least one of: the real- world image, a previously-captured real-world image, satisfies a quality criteria, wherein the previously-captured image is stored at a data repository (
Pizzocchero teaches at Paragraph 0052 that, the physical document and mobile computing device 102 remains stationary during video capture but one or more operational parameters and/or capture settings of image capture device 103 (e.g., flash intensity, flash duration, shutter speed, ISO speed, gain, aperture, light balance, etc.) are modified or adjusted for different frames of the video, in order to cause one or more security features (such as OVDs) on the physical document to become visible or invisible, change color, change appearance, and so forth. In some embodiments, mobile computing device 102 analyzes frames of the video as the frames are being captured and automatically adjusts operational parameters and/or capture settings of image capture device 103 to generate a set of frames with varying imaging conditions, lighting conditions, and/or image characteristics.
Pizzocchero teaches at Paragraph [0059] that as document detection and tracking module 105a tracks the physical document throughout the one or more images, module 105a also assesses imaging conditions in the images in order to dynamically adjust (step 202b) one or more operational parameters of image capture device 103 based upon one or more imaging conditions associated with the physical document, as detected in one or more images of the sequence of images. In some embodiments, document detection and tracking module 105a compares imaging conditions such as lighting characteristics of the background in the image with lighting characteristics of the document and adjusts operational parameters of image capture device 103 based upon the comparison. For example, if the background of the image is very bright and the document is dark relative to the background, document detection and tracking module 105a can adjust exposure settings of image capture device 103 to ensure that the maximum possible image signal is acquired from the document. A variety of different approaches can be used by document detection and tracking module 105a to adjust operational parameters of image capture device 103, such as 1) a rule-based approach (e.g., if background and/or document brightness falls within a range of values and/or a threshold value, adjust exposure settings accordingly to maximize signal from the document); 2) a machine learning model trained on a labelled data set; and/or 3) an end-to-end regression model trained on data. Each of these approaches is described in more detail below.
Pizzocchero teaches at Paragraph [0060] Rule-Based Approach: In some embodiments, the rule-based approach leverages heuristics to define capture settings of image capture device 103 given a set of assessed input criteria. An exemplary set of assessed and defined input criteria are as follows: [0061] If ambient light is too bright, module 105a can adjust image capture device 103 parameters to reduce exposure setting and gain; Pizzocchero teaches at Paragraph [0062] If conditions are too dark, module 105a can instruct image capture device 103 to capture subsequent/additional frames using increasing flash intensity, and/or increase exposure settings of image capture device 103; [0063] If there is glare present on the document in the frame, module 105a can reduce exposure settings of image capture device 103 and/or reduce gain parameters for image capture device 103);
when it is determined that the representation of the at least one region in the at least one of: the real-world image, the previously-captured real-world image, fails to satisfy the quality criteria, controlling the at least one camera for capturing a reference real-world image representing the at least one region, by adjusting the camera settings such that said representation fulfills the quality criteria, while maintaining the pose of the at least one camera at the time of capturing the real-world image with the representation (
At the time of capturing the real-world image, the capture setting of Pizzocchero is maintained and is not associated with the pose of the image capture device 102 without rotation/tilt of the image capture device 102 for a single frame. The capture setting is the flash intensity, flash duration, shutter speed, ISO speed, gain, aperture, light balance.
Pizzocchero teaches at Paragraph 0052 that, the physical document and mobile computing device 102 remains stationary during video capture but one or more operational parameters and/or capture settings of image capture device 103 (e.g., flash intensity, flash duration, shutter speed, ISO speed, gain, aperture, light balance, etc.) are modified or adjusted for different frames of the video, in order to cause one or more security features (such as OVDs) on the physical document to become visible or invisible, change color, change appearance, and so forth. In some embodiments, mobile computing device 102 analyzes frames of the video as the frames are being captured and automatically adjusts operational parameters and/or capture settings of image capture device 103 to generate a set of frames with varying imaging conditions, lighting conditions, and/or image characteristics.
Pizzocchero teaches at Paragraph [0064] that Machine Learning Model Trained on Labelled Data Set: In some embodiments, the approach using a machine learning (ML) model trained on a labelled data set moves beyond the simple heuristics of the rule-based approach to utilize deep learning to convert certain lighting characteristics of the incoming frame(s) into multidimensional embeddings and feed the embeddings to a trained classification model executed by module 105a which evaluates the embeddings using weights adjusted for frames taken with known capture settings to determine whether the incoming frame(s) have sufficient lighting parameters or not to be usable for document verification. In this approach, the classification model can determine one or more parameter adjustments for image capture device 103 and module 105a then adjusts capture parameters for subsequent frames to achieve image capture that falls within acceptable lighting conditions.
Pizzocchero teaches at Paragraph [0065] End-to-End Regression Model: In some embodiments, the approach using an end-to-end regression model trained on data enables the most effective control of the scene and capture settings of image capture device 103. Module 105a executes a trained deep learning regression model to perform end-to-end regression of the lighting conditions and capture settings given any scene, and the regression model can optimize for the specifics of the scene so as to maximize the signal acquired from the document and/or OVD while suppressing noise due to visual/optical phenomena.
Pizzocchero teaches at Paragraph [0066] that document detection and tracking module 105a also assesses physical properties of the document in the images in order to adjust operational parameters and/or capture settings of image capture device 103 and document detection and tracking module 105a can adjust operational parameters and/or capture settings of image capture device 103 to ensure that the maximum possible image signal is acquired from the document. For example, document detection and tracking module 105a can be configured to utilize a deep learning classification model that is trained on surface properties of different materials in images to evaluate the incoming frames, classify a likely composition/material of the document depicted in the frames, and adjust operational parameters to adjust capture settings);
generating training data comprising reference data and input data, wherein the reference data comprises the reference real-world image representing the at least one region, and the input data comprises the at least one of: the real-world image of the at least one region, the previously-captured real-world image of the at least one region (
Pizzocchero’s images taken with know capture settings are used as input images (input data) representing the at least document region to adjust the weights of the machine learning model in order to determine the quality metrics of the incoming images (reference data) representing the least document region.
Pizzocchero teaches at Paragraph [0064] that Machine Learning Model Trained on Labelled Data Set: In some embodiments, the approach using a machine learning (ML) model trained on a labelled data set moves beyond the simple heuristics of the rule-based approach to utilize deep learning to convert certain lighting characteristics of the incoming frame(s) (reference frames) into multidimensional embeddings and feed the embeddings to a trained classification model executed by module 105a which evaluates the embeddings using weights adjusted for frames (input data comprising the previously captured real-world image of the document region) taken with known capture settings to determine whether the incoming frame(s) (input data comprising the real-world image data) have sufficient lighting parameters or not to be usable for document verification. In this approach, the classification model can determine one or more parameter adjustments for image capture device 103 and module 105a then adjusts capture parameters for subsequent frames (reference data comprising the real-time captured real-world image) to achieve image capture that falls within acceptable lighting conditions.
Pizzocchero teaches at Paragraph [0066] that document detection and tracking module 105a also assesses physical properties of the document in the images (the input data) in order to adjust operational parameters and/or capture settings of image capture device 103 and document detection and tracking module 105a can adjust operational parameters and/or capture settings of image capture device 103 to ensure that the maximum possible image signal is acquired from the document. For example, document detection and tracking module 105a can be configured to utilize a deep learning classification model that is trained on surface properties of different materials in images (input images) to evaluate the incoming frames (reference frames), classify a likely composition/material of the document depicted in the frames, and adjust operational parameters to adjust capture settings); and
sending the training data to a processor that is configured to train a first neural network for generating real-world images that satisfy the quality criteria by processing real-world images that fail to satisfy the quality criteria (
Pizzocchero teaches at Paragraph [0064] that Machine Learning Model Trained on Labelled Data Set: In some embodiments, the approach using a machine learning (ML) model trained on a labelled data set moves beyond the simple heuristics of the rule-based approach to utilize deep learning to convert certain lighting characteristics of the incoming frame(s) into multidimensional embeddings and feed the embeddings to a trained classification model executed by module 105a which evaluates the embeddings using weights adjusted for frames taken with known capture settings to determine whether the incoming frame(s) have sufficient lighting parameters or not to be usable for document verification. In this approach, the classification model can determine one or more parameter adjustments for image capture device 103 and module 105a then adjusts capture parameters for subsequent frames to achieve image capture that falls within acceptable lighting conditions).
It would have been obvious to one of the ordinary skill in the art before the filing date of the instant application to have incorporated Pizzocchero’s machine learning model to have modified Agrawal’s camera control module (CCM) for controlling the at one front facing camera and the at least one rear facing camera to have performed one or more parameter adjustments for image capture device 103 and to have allowed the module 105a to adjust capture parameters for subsequent frames to achieve image capture that falls within acceptable lighting conditions according to Pizzocchero’s machine learning model. One of the ordinary skill in the art would have been motivated to have provided camera control module for controlling the camera setting according to Agrawal based on the eye gaze tracking and the camera pose tracking to have focused on the region of interest and for controlling the camera setting according to Pizzochero to achieve image capture that falls within acceptable lighting conditions.
Rao explicitly teaches the claim limitation:
determining whether a representation of the at least one region in at least one of: the real- world image, a previously-captured real-world image, satisfies a quality criteria, wherein the previously-captured image is stored at a data repository (Rao teaches Page 13, column 2, lines 13-16 that the camera parameter settings are tuned responsive to a perceptual no-reference quality determination indicating a quality of the captured frames being below a quality threshold level);
when it is determined that the representation of the at least one region in the at least one of: the real-world image, the previously-captured real-world image, fails to satisfy the quality criteria, controlling the at least one camera for capturing a reference real-world image representing the at least one region, by adjusting the camera settings such that said representation fulfills the quality criteria, while maintaining the pose of the at least one camera at the time of capturing the real-world image with the representation (
At the time of capturing the video, the pose of the video camera is maintained and camera parameters are not associated with the pose of the image capture device 102 without rotation/tilt of the video camera for a single frame. The capture setting is the brightness, color, contrast and sharpness (see Rao Paragraph 0024) while the camera system 202 of FIG. 2 is in the fixed position (fixed pose).
Rao teaches at Paragraph 0028 that the adaptive perceptual camera tuning (APT) system can utilize a neural network and reinforcement learning (RL) to automatically and adaptively tune camera parameters remotely to generate a high-quality video feed and at Paragraph 0096 that a video/video stream can be acquired using one or more video cameras deployed in various environmental settings in block 502 for use (e.g., real-time use) in video analytics applications (e.g., object/person detection).
Rao teaches Page 13, column 2, lines 13-16 that the camera parameter settings are tuned responsive to a perceptual no-reference quality determination indicating a quality of the captured frames being below a quality threshold level.
Rao teaches at Paragraph 0076 that reinforcement learning using a DNN can be executed in block 220 to automatically adapt the camera parameters to improve the accuracy of AUs in accordance with aspects of the present invention and at Paragraph 0077 that reinforcement learning (RL) can be utilized in block 220 to determine the best camera settings for a particular scene to provide optimal AU accuracy for video analytics tasks. This learning can be performed in an online manner using RL, in which the system 200 can learn the best camera settings in real-time in any of a plurality of environmental conditions.
Rao teaches at Paragraph 0097 that camera parameters (e.g., brightness, color, contrast, and sharpness), can be dynamically and automatically adjusted based on the reward function derived from the perceptual quality estimator.);
generating training data comprising reference data and input data, wherein the reference data comprises the reference real-world image representing the at least one region, and the input data comprises the at least one of: the real-world image of the at least one region, the previously-captured real-world image of the at least one region (Rao teaches at Paragraph 0097 that camera parameters (e.g., brightness, color, contrast, and sharpness), can be dynamically and automatically adjusted based on the reward function derived from the perceptual quality estimator.
Rao teaches at Paragraph 0076 that reinforcement learning using a DNN can be executed in block 220 to automatically adapt the camera parameters to improve the accuracy of AUs in accordance with aspects of the present invention and at Paragraph 0077 that reinforcement learning (RL) can be utilized in block 220 to determine the best camera settings for a particular scene to provide optimal AU accuracy for video analytics tasks. This learning can be performed in an online manner using RL, in which the system 200 can learn the best camera settings in real-time in any of a plurality of environmental conditions.
Rao teaches at Paragraph 0028 that the adaptive perceptual camera tuning (APT) system can utilize a neural network and reinforcement learning (RL) to automatically and adaptively tune camera parameters remotely to generate a high-quality video feed and at Paragraph 0096 that a video/video stream can be acquired using one or more video cameras deployed in various environmental settings in block 502 for use (e.g., real-time use) in video analytics applications (e.g., object/person detection). A sequence of video frames captured using a camera can be monitored, captured, and/or received (e.g., by a server 206 with reference to FIG. 2) in block 502. In some embodiments, at 504, the method can include dynamically determining optimal camera parameters to capture superior-quality video frames in response to changing environmental conditions and scene context using Reinforcement Learning (RL) and a Convolutional Neural Network (CNN) in accordance with aspects of the present invention. At 506, the method can include processing the captured video frames with an analytics unit to generate insights, including identifying specific objects, tracking behaviors, and detecting anomalies using RL. In block 508, the RL can be integrated with a perceptual quality estimator (as a reward function) based on a convolutional neural network (CNN), where the estimator can assess video frame quality based on human perceptual aspects); and
sending the training data to a processor that is configured to train a first neural network for generating real-world images that satisfy the quality criteria by processing real-world images that fail to satisfy the quality criteria (Rao teaches at Paragraph 0097 that camera parameters (e.g., brightness, color, contrast, and sharpness), can be dynamically and automatically adjusted based on the reward function derived from the perceptual quality estimator. Rao teaches at Paragraph 0096 that a video/video stream can be acquired using one or more video cameras deployed in various environmental settings in block 502 for use (e.g., real-time use) in video analytics applications (e.g., object/person detection). A sequence of video frames captured using a camera can be monitored, captured, and/or received (e.g., by a server 206 with reference to FIG. 2) in block 502. In some embodiments, at 504, the method can include dynamically determining optimal camera parameters to capture superior-quality video frames in response to changing environmental conditions and scene context using Reinforcement Learning (RL) and a Convolutional Neural Network (CNN) in accordance with aspects of the present invention. At 506, the method can include processing the captured video frames with an analytics unit to generate insights, including identifying specific objects, tracking behaviors, and detecting anomalies using RL. In block 508, the RL can be integrated with a perceptual quality estimator (as a reward function) based on a convolutional neural network (CNN), where the estimator can assess video frame quality based on human perceptual aspects.
Rao teaches Page 13, column 2, lines 13-16 that the camera parameter settings are tuned responsive to a perceptual no-reference quality determination indicating a quality of the captured frames being below a quality threshold level).
It would have been obvious to one of the ordinary skill in the art before the filing date of the instant application to have incorporated Rao’s CNN to have modified Agrawal’s camera control module (CCM) for controlling the at one front facing camera and the at least one rear facing camera to have tuned the camera parameter settings responsive to a perceptual no-reference quality determination indicating a quality of the captured frames being below a quality threshold level according to Rao’s CNN. One of the ordinary skill in the art would have been motivated to have provided camera control module for controlling the camera setting based on the eye gaze tracking and the camera pose tracking to have focused on the region of interest according to Agrawal and for controlling the camera setting according to Rao responsive to a perceptual no-reference quality determination indicating a quality of the captured frames being below a quality threshold level.
Grant explicitly teaches the claim limitation:
determining whether a representation of the at least one region in at least one of: the real- world image, a previously-captured real-world image, satisfies a quality criteria, wherein the previously-captured image is stored at a data repository (Grant teaches at Paragraph 0058 that the image analysis application determines resolution adjustment patterns among respective discrete objects in one or more previously captured images by applying at least one classification machine learning algorithm and/or at least one clustering machine learning algorithm. The image analysis application determines type and/or degree of resolution adjustment (e.g., type and/or degree of a blurring effect) among respective discrete objects by applying a classification algorithm. Additionally or alternatively, the image analysis application determines level of resolution adjustment among respective discrete objects by applying a quality threshold (QT) clustering algorithm);
when it is determined that the representation of the at least one region in the at least one of: the real-world image, the previously-captured real-world image, fails to satisfy the quality criteria, controlling the at least one camera for capturing a reference real-world image representing the at least one region, by adjusting the camera settings such that said representation fulfills the quality criteria, while maintaining the pose of the at least one camera at the time of capturing the real-world image with the representation (
In other words, increasing resolution or increasing focus in Grant involves changing the zoom level or changing the focal length (while maintaining the pose of the image capture device) and does not involve changing the image capture parameter such as the pose of the image capture device.
Grant teaches at Paragraph [0049] the sensor data pertains to environmental factors within a predefined vicinity of image capture, e.g., weather or air quality, based upon which the image analysis application may facilitate further adjustment. For instance, the image analysis application may adjust contrast of the at least one captured image and/or the plurality of discrete objects therein upon determining via sensor data that the weather is cloudy. In another instance, the image analysis application may adjust color of the at least one captured image and/or the plurality of discrete objects therein upon determining via sensor data that the air quality is low due to smoke.
Grant teaches at Paragraph [0066] that he emphasis of one discrete object relative to the others may entail emphasizing one or more photographic parameters of such discrete object (e.g., increasing resolution, increasing focus, increasing contrast, etc.). Per step 230, the image analysis application presents to Client A the captured image modification options, e.g., via a viewfinder of the image capture device. Accordingly, Client A may select one or more of the captured image modification options, depending upon whether Client A prefers emphasis of the oak tree, emphasis of the mountain, or (optionally) emphasis of both the oak tree and the mountain in the at least one captured image.
Grant teaches at Paragraph 0050 that the image analysis application may facilitate adjustment of the pre-capture image parameters of the image capture device such that the image capture device focuses upon, zooms in upon, and/or otherwise emphasizes the tiger for purposes of subsequent image capture. The image analysis application optionally facilitates adjustment of at least one pre-capture image parameter in view of the image capture learning model prior to capture of a subsequent image based upon data collected by the at least one monitoring sensor. For instance, assuming that the client discusses a deer while sitting in a car and aims the image capture device toward the deer, based upon sensor data collected pertaining to the client deer discussion, the image analysis application optionally facilitates adjustment of at least one pre-capture image parameter such that the image capture device focuses upon, zooms in upon, and/or otherwise emphasizes the deer for purposes of subsequent image capture.
Grant teaches at Paragraph 0058 that the image analysis application determines resolution adjustment patterns among respective discrete objects in one or more previously captured images by applying at least one classification machine learning algorithm and/or at least one clustering machine learning algorithm. The image analysis application determines type and/or degree of resolution adjustment (e.g., type and/or degree of a blurring effect) among respective discrete objects by applying a classification algorithm. Additionally or alternatively, the image analysis application determines level of resolution adjustment among respective discrete objects by applying a quality threshold (QT) clustering algorithm);
generating training data comprising reference data and input data, wherein the reference data comprises the reference real-world image representing the at least one region, and the input data comprises the at least one of: the real-world image of the at least one region, the previously-captured real-world image of the at least one region (Grant teaches at Paragraph 0058 that the image analysis application applies at least one machine learning algorithm based upon the parsed metadata derived from the archived image capture data and the collected profile data associated with the client and at Paragraph 0061 that image analysis application identifies any such representation based upon audiovisual processing of one or more of the plurality of contextual inputs, e.g., processing of visual and/or audio aspects associated with one or more recently captured images.
Grant teaches at Paragraph 0058 that the image analysis application determines contextual details associated with the archived image capture data and/or the collected profile data associated with the client by applying at least one audiovisual processing machine learning algorithm to inputs derived from parsed image metadata (e.g., associated with one or more previously captured images) and/or parsed audiovisual metadata. According to such further embodiment, the image analysis application identifies respective discrete objects in one or more previously captured images by applying at least one object detection algorithm. One object detection algorithm option is a R-CNN algorithm. The image analysis application optionally determines whether a respective discrete object in one or more previously captured images is visible (e.g., in terms of resolution) based upon application of a R-CNN algorithm in conjunction with analysis of any Boolean metadata pertaining to the respective discrete object. Another object detection algorithm option is a YOLO algorithm. The image analysis application determines resolution adjustment patterns among respective discrete objects in one or more previously captured images by applying at least one classification machine learning algorithm and/or at least one clustering machine learning algorithm. The image analysis application determines type and/or degree of resolution adjustment (e.g., type and/or degree of a blurring effect) among respective discrete objects by applying a classification algorithm. Additionally or alternatively, the image analysis application determines level of resolution adjustment among respective discrete objects by applying a quality threshold (QT) clustering algorithm); and
sending the training data to a processor that is configured to train a first neural network for generating real-world images that satisfy the quality criteria by processing real-world images that fail to satisfy the quality criteria (Grant teaches at Paragraph 0058 that the image analysis application determines resolution adjustment patterns among respective discrete objects in one or more previously captured images by applying at least one classification machine learning algorithm and/or at least one clustering machine learning algorithm. The image analysis application determines type and/or degree of resolution adjustment (e.g., type and/or degree of a blurring effect) among respective discrete objects by applying a classification algorithm. Additionally or alternatively, the image analysis application determines level of resolution adjustment among respective discrete objects by applying a quality threshold (QT) clustering algorithm.
Grant teaches at Paragraph 0065 that the emphasis of one discrete object relative to the others may entail emphasizing one or more photographic parameters of such discrete object (e.g., increasing resolution, increasing focus, increasing contrast, etc.). Per step 230, the image analysis application presents to Client A the captured image modification options, e.g., via a viewfinder of the image capture device. Accordingly, Client A may select one or more of the captured image modification options, depending upon whether Client A prefers emphasis of the oak tree, emphasis of the mountain, or (optionally) emphasis of both the oak tree and the mountain in the at least one captured image.
Grant teaches at Paragraph [0066] that, per step 235 the image analysis application may facilitate at least one further photographic adjustment to the at least one captured image in the context of the example scenario, e.g., based upon at least one comment received from Client A and/or based upon data collected by at least one monitoring sensor associated with the image capture device of Client A. Optionally, per step 240, the image analysis application may facilitate at least one adjustment for subsequent image capture. For instance, the image analysis application may facilitate adjustment of at least one pre-capture image parameter in view of the image capture learning model and/or may facilitate adjustment of at least one element of photographic equipment (e.g., a gimbal) in view of the image capture learning model prior to capture of a subsequent image via the image capture device of Client A.
Grant teaches at Paragraph 0046 that the image analysis application provides the client an option to adjust post-capture image parameters with respect to the at least one captured image in view of the image capture learning model. In the context of the various embodiments, post-capture image parameters pertain to digital image adjustments made following capture.
Grant teaches at Paragraph 0051 that the image analysis application facilitates adjustment of at least one element of photographic equipment at step 240 in view of the image capture learning model prior to capture of a subsequent image via the image capture device. According to such further embodiment, the image analysis application optionally facilitates adjustment of a gimbal and/or other photographic stabilization component attached to or otherwise associated with the image capture device. The image analysis application optionally relays at least one control signal to the gimbal and/or other photographic stabilization component in order to facilitate such adjustment. Additionally or alternatively, based upon the image capture learning model, the image analysis application optionally facilitates execution of autonomous or semi-autonomous adjustment of photographic equipment for subsequent image capture. Additionally or alternatively, based upon the image capture learning model, the image analysis application provides to the client at least one photographic equipment adjustment instruction, e.g., via the client interface of the image capture device and/or via another client communication channel.).
It would have been obvious to one of the ordinary skill in the art before the filing date of the instant application to have incorporated Grant’s machine learning model to have modified Agrawal’s camera control module (CCM) for controlling the at one front facing camera and the at least one rear facing camera to have performed by the image analysis application to facilitate at least one adjustment for subsequent image capture according to Grant’s machine learning model. One of the ordinary skill in the art would have been motivated to have provided camera control module for controlling the camera setting according to Agrawal based on the eye gaze tracking and the camera pose tracking to have focused on the region of interest and for controlling the camera setting according to Grant by the image analysis application to facilitate at least one adjustment for subsequent image capture.
Re Claim 2:
The claim 2 encompasses the same scope of invention as that of the claim 1 except additional claim limitation that the quality criteria comprises at least one of: absence of defocus blur; absence of motion blur; absence of saturation; absence of noise; a spatial resolution being higher than a predefined threshold.
Pizzocchero further teaches the claim limitation that the quality criteria comprises at least one of: absence of defocus blur; absence of motion blur; absence of saturation; absence of noise; a spatial resolution being higher than a predefined threshold (
Pizzocchero teaches at Paragraph [0014] that the one or more quality metrics comprise (i) global image quality metrics including one or more of: glare, blur, white balance, or sensor noise characteristics, (ii) local image quality metrics including one or more of: blur, sharpness, text region confidence, character confidence, or edge detection, or (iii) both the global image quality metrics and the local image quality metrics.
Pizzocchero teaches at Paragraph [0051] that mobile computing device 102 can detect baseline imaging conditions (e.g., light intensity, glare, blur, white balance, sensor noise characteristics such as blooming, readout noise, or custom calibration variations, focus, etc.) and/or changes in imaging conditions associated with the physical document and adjust operational parameters of image capture device 103 (e.g., flash, aperture, pixel gain, etc.) accordingly as will be described in detail herein.
Pizzocchero teaches at Paragraph [0062] If conditions are too dark, module 105a can instruct image capture device 103 to capture subsequent/additional frames using increasing flash intensity, and/or increase exposure settings of image capture device 103; [0063] If there is glare present on the document in the frame, module 105a can reduce exposure settings of image capture device 103 and/or reduce gain parameters for image capture device 103.
Pizzocchero teaches at Paragraph [0059] that as document detection and tracking module 105a tracks the physical document throughout the one or more images, module 105a also assesses imaging conditions in the images in order to dynamically adjust (step 202b) one or more operational parameters of image capture device 103 based upon one or more imaging conditions associated with the physical document, as detected in one or more images of the sequence of images. In some embodiments, document detection and tracking module 105a compares imaging conditions such as lighting characteristics of the background in the image with lighting characteristics of the document and adjusts operational parameters of image capture device 103 based upon the comparison. For example, if the background of the image is very bright and the document is dark relative to the background, document detection and tracking module 105a can adjust exposure settings of image capture device 103 to ensure that the maximum possible image signal is acquired from the document. A variety of different approaches can be used by document detection and tracking module 105a to adjust operational parameters of image capture device 103, such as 1) a rule-based approach (e.g., if background and/or document brightness falls within a range of values and/or a threshold value, adjust exposure settings accordingly to maximize signal from the document); 2) a machine learning model trained on a labelled data set; and/or 3) an end-to-end regression model trained on data. Each of these approaches is described in more detail below).
Re Claim 3:
The claim 3 encompasses the same scope of invention as that of the claim 1 except additional claim limitation that the camera settings comprise at least one of: a focus distance, an exposure, a white balance, of the at least one camera.
Pizzocchero further teaches the claim limitation that the camera settings comprise at least one of: a focus distance, an exposure, a white balance, of the at least one camera (
Pizzocchero teaches at Paragraph [0108] that the physical document and mobile computing device 102 each remains stationary relative to each other, while certain operational elements of image capture device 103 (e.g., flash, exposure, focus, white balance, gain and offset, etc.) are dynamically adjusted after each image capture using a feedback loop, so that different frames of the video capture the physical document under a variety of lighting conditions and capture conditions (e.g., exposure, aperture, gain, etc.).
Pizzocchero teaches at Paragraph [0111] that SDK 105 instructs image capture device 103 to capture (step 802) images of the physical document during which SDK 105 adjusts (step 802a) one or more operational parameters of image capture device 103—which results in different frames of the video having different capture settings including but not limited to: gain settings, offset, exposure settings, focus values, aperture values, lighting changes, flash intensity.
Pizzocchero teaches at Paragraph [0114] that the video capture process is passive for the user because the physical document remains stationary, while mobile computing device 102 adjusts one or more operational parameters of image capture device 103, resulting in a sequence of images captured from a single perspective (e.g., in a flat plane without any three-dimensional rotation or tilting) but using varying capture settings (e.g., lighting, aperture, focus, exposure, gain, etc.).
Pizzocchero teaches at Paragraph [0014] that the one or more quality metrics comprise (i) global image quality metrics including one or more of: glare, blur, white balance, or sensor noise characteristics, (ii) local image quality metrics including one or more of: blur, sharpness, text region confidence, character confidence, or edge detection, or (iii) both the global image quality metrics and the local image quality metrics.
Pizzocchero teaches at Paragraph [0051] that mobile computing device 102 can detect baseline imaging conditions (e.g., light intensity, glare, blur, white balance, sensor noise characteristics such as blooming, readout noise, or custom calibration variations, focus, etc.) and/or changes in imaging conditions associated with the physical document and adjust operational parameters of image capture device 103 (e.g., flash, aperture, pixel gain, etc.) accordingly as will be described in detail herein.
Pizzocchero teaches at Paragraph [0062] If conditions are too dark, module 105a can instruct image capture device 103 to capture subsequent/additional frames using increasing flash intensity, and/or increase exposure settings of image capture device 103; [0063] If there is glare present on the document in the frame, module 105a can reduce exposure settings of image capture device 103 and/or reduce gain parameters for image capture device 103.
Pizzocchero teaches at Paragraph [0059] that as document detection and tracking module 105a tracks the physical document throughout the one or more images, module 105a also assesses imaging conditions in the images in order to dynamically adjust (step 202b) one or more operational parameters of image capture device 103 based upon one or more imaging conditions associated with the physical document, as detected in one or more images of the sequence of images. In some embodiments, document detection and tracking module 105a compares imaging conditions such as lighting characteristics of the background in the image with lighting characteristics of the document and adjusts operational parameters of image capture device 103 based upon the comparison. For example, if the background of the image is very bright and the document is dark relative to the background, document detection and tracking module 105a can adjust exposure settings of image capture device 103 to ensure that the maximum possible image signal is acquired from the document. A variety of different approaches can be used by document detection and tracking module 105a to adjust operational parameters of image capture device 103, such as 1) a rule-based approach (e.g., if background and/or document brightness falls within a range of values and/or a threshold value, adjust exposure settings accordingly to maximize signal from the document); 2) a machine learning model trained on a labelled data set; and/or 3) an end-to-end regression model trained on data. Each of these approaches is described in more detail below).
Re Claim 4:
The claim 4 encompasses the same scope of invention as that of the claim 1 except additional claim limitation that controlling the at least one camera for capturing the reference real-world image comprises: determining reference values of the camera settings based on at least one of: an optical depth of the at least one region, lighting conditions in the at least one region, such that the reference values, when employed, enable the representation of the at least one region in the reference real- world image to satisfy the quality criteria; and generating a control signal for the at least one camera to employ the determined reference values of the camera settings, for capturing the at least one reference real-world image.
Pizzocchero teaches the claim limitation that controlling the at least one camera for capturing the reference real-world image comprises: determining reference values of the camera settings based on at least one of: an optical depth of the at least one region, lighting conditions in the at least one region, such that the reference values, when employed, enable the representation of the at least one region in the reference real-world image to satisfy the quality criteria; and generating a control signal for the at least one camera to employ the determined reference values of the camera settings, for capturing the at least one reference real-world image (
Pizzocchero teaches at Paragraph 0052 that the capture settings of image capture device 103 includes flash intensity, flash duration, shutter speed, ISO speed, gain, aperture, light balance which can be modified or adjusted for different frames.
Pizzocchero teaches at Paragraph [0064] that Machine Learning Model Trained on Labelled Data Set: In some embodiments, the approach using a machine learning (ML) model trained on a labelled data set moves beyond the simple heuristics of the rule-based approach to utilize deep learning to convert certain lighting characteristics of the incoming frame(s) into multidimensional embeddings and feed the embeddings to a trained classification model executed by module 105a which evaluates the embeddings using weights adjusted for frames taken with known capture settings (reference settings) to determine whether the incoming frame(s) have sufficient lighting parameters or not to be usable for document verification. In this approach, the classification model can determine one or more parameter adjustments for image capture device 103 and module 105a then adjusts capture parameters for subsequent frames to achieve image capture that falls within acceptable lighting conditions).
Re Claim 5:
The claim 5 encompasses the same scope of invention as that of the claim 1 except additional claim limitation that receiving, from the processor, weights of the first neural network that are learnt upon the training of the first neural network at the processor; transferring learning of the first neural network to a second neural network, by applying the weights to the second neural network; and processing real-world images that do not satisfy the quality criteria, captured by the at least one camera after the step of transferring learning, for generating corresponding real-world images that satisfy the quality criteria, by employing the second neural network.
Pizzocchero teaches the claim limitation that receiving, from the processor, weights of the first neural network that are learnt upon the training of the first neural network at the processor; transferring learning of the first neural network to a second neural network, by applying the weights to the second neural network; and processing real-world images that do not satisfy the quality criteria, captured by the at least one camera after the step of transferring learning, for generating corresponding real-world images that satisfy the quality criteria, by employing the second neural network (
Pizzocchero teaches at Paragraph [0064] that Machine Learning Model Trained on Labelled Data Set: In some embodiments, the approach using a machine learning (ML) model trained on a labelled data set moves beyond the simple heuristics of the rule-based approach to utilize deep learning to convert certain lighting characteristics of the incoming frame(s) into multidimensional embeddings and feed the embeddings to a trained classification model executed by module 105a which evaluates the embeddings using weights adjusted for frames taken with known capture settings (reference settings) to determine whether the incoming frame(s) have sufficient lighting parameters or not to be usable for document verification. In this approach, the classification model can determine one or more parameter adjustments for image capture device 103 and module 105a then adjusts capture parameters for subsequent frames to achieve image capture that falls within acceptable lighting conditions).
Re Claim 6:
The claim 6 encompasses the same scope of invention as that of the claim 1 except additional claim limitation that generating an extended-reality image using the real-world image; and controlling at least one display, for displaying the extended-reality image.
Pizzocchero teaches the claim limitation that generating an extended-reality image using the real-world image; and controlling at least one display, for displaying the extended-reality image (Pizzocchero teaches at FIGS. 13A-13B and Paragraph 0131 that the user interface can then display bounding lines 1350 at the corners of the document and/or a bounding box 1360).
Re Claim 7:
The claim 7 encompasses the same scope of invention as that of the claim 1 except additional claim limitation that generating at least one reprojected real-world image, by time warping the real-world image, during a time period of controlling the at least one camera for capturing the reference real-world image, wherein upon elapsing of said time period, the method further comprises controlling the at least one camera for capturing a next real-world image; generating at least one extended-reality image using the at least one reprojected real-world image; and controlling at least one display, for displaying the at least one extended-reality image until a next extended-reality image that is generated using the next real-world image is generated for displaying.
Pizzocchero teaches the claim limitation that generating at least one reprojected real-world image, by time warping the real-world image, during a time period of controlling the at least one camera for capturing the reference real-world image, wherein upon elapsing of said time period, the method further comprises controlling the at least one camera for capturing a next real-world image; generating at least one extended-reality image using the at least one reprojected real-world image; and controlling at least one display, for displaying the at least one extended-reality image until a next extended-reality image that is generated using the next real-world image is generated for displaying (
0077] FIGS. 4A and 4B comprise an exemplary user interface workflow 400 for guiding a user in tilting and rotating a document during the Active Document Liveness process. As shown in FIG. 4A, a user of mobile computing device 102 can hold a document (e.g., a driver's license) in front of image capture device 103 (see screen 402) and a user interface of device 102 (implemented by SDK 105) can guide the user to align the document with a user interface element (e.g., a circle) so that the document is fully visible and at a predetermined distance from the image capture device 103 (see screen 404).
[0078] The user interface of mobile computing device 102 can then display another user interface element (e.g., bounding lines 450 at the corners of the document and/or a bounding box 460) in the user interface that confirms the document is properly positioned and aligned to the image capture device 103 (see screen 406). The user interface instructs the user to hold the mobile computing device 102 (and/or the document) still for a moment and module 105d performs classification of the document to confirm the document is a U.K. driver's license (see screen 408). Turning to FIG. 4B, the user interface can instruct the user to tilt and/or rotate the document in certain directions (e.g., left, right, upwards, downwards) while image capture device 103 and module 105d capture and process images of the document as described above (see screens 410, 412, 414). In some embodiments, module 105a can enhance user interface in order to provide a visual indicator to user regarding scanning progress and signal capture. For example, one or more sides of the bounding box 460 can change color (e.g., from white to green) as sufficient range(s) of motion on the corresponding side are met. Once module 105d has determined that the user has rotated and/or tilted the document according to a sufficient range of motion and the captured frames are sufficient for document verification, user interface can display indicia to the user that the document is being scanned (see screen 416) and that the document liveness check is complete, indicating the document is authenticated (see screen 418).
[0079] In some embodiments, module 105d dynamically assesses the document while image capture device 103 captures frames and/or video, given the ambient lighting conditions, to guide a user through the minimum amount of rotation and/or tilt for a specific document to ensure that sufficient OVD signal is acquired for purposes of document authentication. As an example, for a particular document type (e.g., U.S. passport), the minimum rotation/tilt might be 15 degrees up and 25 degrees to the right. For a different document type (e.g., U.K. driver's license), the minimum rotation/tilt might be 25 degrees up and 10 degrees to the left. Furthermore, the particular lighting conditions can result in module 105d dynamically adjusting the minimum rotation/tilt values (as the frames are captured) to ensure that sufficient OVD signal is obtained. For example, in circumstances where ambient light is very bright, a user may only need to rotate a California driver's license 15 degrees to the left (instead of 20 degrees to the left in normal lighting conditions). In another example, the ambient light may be very low and the user may need to rotate a California driver's license 30 degrees to the left in order to obtain sufficient OVD signal. Thus, using a dynamic lighting configuration process, in conjunction with known attributes of the detected document type (as generated from the detection and classification of the document described above), module 105d can dynamically adjust the minimum values for rotation/tilt along any axes or in any directions during image capture and processing, so that the user is automatically instructed via a user interface to move the document appropriately to capture sufficient OVD signal. As can be appreciated, the dynamic nature of this process ensures that the full reconstruction is obtained and the maximal amount of signal is elicited for each specific document and document type—in view of the document's characteristics-thus reducing the burden on the user.
Pizzocchero teaches at FIGS. 13A-13B and Paragraph 0131 that the user interface can then display bounding lines 1350 at the corners of the document and/or a bounding box 1360 and at Paragraph [0132] that, image capture device 103 and module 105d capture and process images of the document using varying capture settings. For example, module 105d can instruct image capture device 103 to capture one or more images of the document using a first set of capture settings, e.g., Auto mode (see screen 1310). Module 105d can then activate a lighting element of mobile computing device 103 (e.g., flash in Torch mode) to capture one or more additional images of the document (see screen 1312). Then, the user interface can display indicia to the user that the document is being scanned (see screen 1314) and that the document liveness check is complete, indicating the document is authenticated (see screen 1316).).
Re Claim 8:
The claim 8 encompasses the same scope of invention as that of the claim 1 except additional claim limitation when it is determined that the representation of the at least one region is represented in the at least one of: the real-world image, the previously-captured real-world image, satisfies the quality criteria, controlling the at least one camera for capturing an input real-world image representing the at least one region, by adjusting the camera settings such that said representation fails to satisfy the quality criteria;
generating training data comprising reference data and input data, wherein the reference data comprises the at least one of: the real-world image, the previously-captured real-world image, and the input data comprises the input real-world image; and
sending the training data to a processor that is configured to train a first neural network for generating real-world images that satisfy the quality criteria by processing real-world images that fail to satisfy the quality criteria.
Pizzocchero explicitly teaches the claim limitation:
when it is determined that the representation of the at least one region is represented in the at least one of: the real-world image, the previously-captured real-world image, satisfies the quality criteria, controlling the at least one camera for capturing an input real-world image representing the at least one region, by adjusting the camera settings such that said representation fails to satisfy the quality criteria (
Pizzocchero teaches at Paragraph [0064] that Machine Learning Model Trained on Labelled Data Set: In some embodiments, the approach using a machine learning (ML) model trained on a labelled data set moves beyond the simple heuristics of the rule-based approach to utilize deep learning to convert certain lighting characteristics of the incoming frame(s) into multidimensional embeddings and feed the embeddings to a trained classification model executed by module 105a which evaluates the embeddings using weights adjusted for frames taken with known capture settings to determine whether the incoming frame(s) have sufficient lighting parameters or not to be usable for document verification. In this approach, the classification model can determine one or more parameter adjustments for image capture device 103 and module 105a then adjusts capture parameters for subsequent frames to achieve image capture that falls within acceptable lighting conditions.
Pizzocchero teaches at Paragraph [0065] End-to-End Regression Model: In some embodiments, the approach using an end-to-end regression model trained on data enables the most effective control of the scene and capture settings of image capture device 103. Module 105a executes a trained deep learning regression model to perform end-to-end regression of the lighting conditions and capture settings given any scene, and the regression model can optimize for the specifics of the scene so as to maximize the signal acquired from the document and/or OVD while suppressing noise due to visual/optical phenomena.
Pizzocchero teaches at Paragraph [0066] that document detection and tracking module 105a also assesses physical properties of the document in the images in order to adjust operational parameters and/or capture settings of image capture device 103 and document detection and tracking module 105a can adjust operational parameters and/or capture settings of image capture device 103 to ensure that the maximum possible image signal is acquired from the document. For example, document detection and tracking module 105a can be configured to utilize a deep learning classification model that is trained on surface properties of different materials in images to evaluate the incoming frames, classify a likely composition/material of the document depicted in the frames, and adjust operational parameters to adjust capture settings);
generating training data comprising reference data and input data, wherein the reference data comprises the at least one of: the real-world image, the previously-captured real-world image, and the input data comprises the input real-world image; and
(
Pizzocchero teaches at Paragraph [0064] that Machine Learning Model Trained on Labelled Data Set: In some embodiments, the approach using a machine learning (ML) model trained on a labelled data set moves beyond the simple heuristics of the rule-based approach to utilize deep learning to convert certain lighting characteristics of the incoming frame(s) into multidimensional embeddings and feed the embeddings to a trained classification model executed by module 105a which evaluates the embeddings using weights adjusted for frames taken with known capture settings to determine whether the incoming frame(s) have sufficient lighting parameters or not to be usable for document verification. In this approach, the classification model can determine one or more parameter adjustments for image capture device 103 and module 105a then adjusts capture parameters for subsequent frames to achieve image capture that falls within acceptable lighting conditions); and
sending the training data to a processor that is configured to train a first neural network for generating real-world images that satisfy the quality criteria by processing real-world images that fail to satisfy the quality criteria (
Pizzocchero teaches at Paragraph [0064] that Machine Learning Model Trained on Labelled Data Set: In some embodiments, the approach using a machine learning (ML) model trained on a labelled data set moves beyond the simple heuristics of the rule-based approach to utilize deep learning to convert certain lighting characteristics of the incoming frame(s) into multidimensional embeddings and feed the embeddings to a trained classification model executed by module 105a which evaluates the embeddings using weights adjusted for frames taken with known capture settings to determine whether the incoming frame(s) have sufficient lighting parameters or not to be usable for document verification. In this approach, the classification model can determine one or more parameter adjustments for image capture device 103 and module 105a then adjusts capture parameters for subsequent frames to achieve image capture that falls within acceptable lighting conditions).
Re Claim 9:
The claim 9 encompasses the same scope of invention as that of the claim 1 except additional claim limitation that determining the spatial geometry of the real-world environment, by processing sensor data that is collected by at least one depth sensor.
Agrawal further teaches the claim limitation that determining the spatial geometry of the real-world environment, by processing sensor data that is collected by at least one depth sensor
(Agrawal teaches at Paragraph [0070] In one embodiment, adjusting the active rear facing camera using the generated new camera settings 266 can include adjusting a directional setting (pose) of the rear facing camera 133A to focus the FOV 362 of rear facing camera 133A on the new second ROI 250B that is determined from the current eye gaze location (i.e., second location 262B). In another embodiment, adjusting the active rear facing camera using the generated new camera settings 266 can include adjusting a zoom level of the rear facing camera 133A to magnify or reduce captured images of the new ROI (i.e., second ROI 250B). In an additional embodiment, adjusting the active rear facing camera using the generated new camera settings 266 can include adjusting a focal distance of the rear facing camera 133A to focus the rear facing camera on the new ROI (i.e., second ROI 250B).
Agrawal teaches at Paragraph 0055 that eye gaze location 262 is a specific location that a user of electronic device 100 is looking, which correlates to and is determined based on the eye gaze direction 260. Eye gaze location 262 can include a plurality of discrete locations, such as first location 262A, second location 262B, and third location 262C. According to one embodiment, eye gaze location 262 can include the orientation to the location where the user is looking based on the corresponding eye gaze direction. For example, eye gaze location 262 can have a vertical orientation of 5 degrees upwards, and a horizontal direction that is positive (right) 15 degrees from the front surface 176 of electronic device 100. It is appreciated that the dimensions and orientations measurements would be based on the front surface of electronic device 100 representing a vertical/Y plane, in a 3-dimensional X-Y-Z axes coordinate plane extending through a center point of the front surface of electronic device 100.
Agrawal teaches at Paragraph [0062] FIG. 3A illustrates electronic device 100 being used by a user 310 to record or capture video data 270. A cat 324 and a dog 326 are located behind rear surface 180 away from electronic device 100. The face 314 of the user 310 includes a pair of eyes 316 that are looking in first eye gaze direction 260A. In FIG. 3A, the user is looking at cat 324 and dog 326 along first eye gaze direction 260A towards first ROI 250A. First ROI 250A includes cat 324 and dog 326. Cat 324 and dog 326 are within FOV 346 of the active camera (i.e., rear facing main camera 133A).
).
Re Claim 10:
The claim 10 recites a display apparatus comprising: a gaze-tracking means; a pose-tracking means; at least one camera; and at least one processor configured to:
determine a gaze point and a gaze depth of a user's eyes, by processing gaze-tracking data that is collected by the gaze-tracking means;
control the at least one camera to capture a real-world image of a real-world environment, by adjusting camera settings according to the gaze point and the gaze depth;
determine a pose of the at least one camera at a time of capturing the real-world image, by processing pose-tracking data that is collected by the pose-tracking means;
identify at least one region of the real-world environment that is represented in the real- world image, based on a spatial geometry of the real-world environment and the pose of the at least one camera;
determine whether a representation of the at least one region in at least one of: the real- world image, a previously-captured real-world image, satisfies a quality criteria, wherein the previously-captured image is stored at a data repository that is communicably coupled with the at least one processor;
when it is determined that the representation of the at least one region in the at least one of: the real-world image, the previously-captured real-world image, fails to satisfy the quality criteria, control the at least one camera to capture a reference real-world image representing the at least one region, by adjusting the camera settings such that said representation fulfills the quality criteria;
generate training data comprising reference data and input data, wherein the reference data comprises the reference real-world image, and the input data comprises the at least one of: the real- world image, the previously-captured real-world image; and
send the training data to a processor that is configured to train a first neural network to generate real-world images that satisfy the quality criteria by processing real-world images that fail to satisfy the quality criteria.
The claim 10 is in parallel with the claim 1 in an apparatus form. The claim 10 is subject to the same rationale of rejection as the claim 1.
In other words, Agrawal teaches a display apparatus comprising (e.g., the electronic device 100 of FIG. 1 including the touch screen interface 131): a gaze-tracking means (Agrawal teaches at FIG. 4A-4C and Paragraph 0049 that CCM 136 includes program code that is executed by processor 102 to enable electronic device 100 to perform the various features of the present disclosure. In one or more embodiments, CCM 136 enables electronic device 100 to track an eye gaze direction of a user within a display and to automatically adjust an active camera based on the eye gaze tracking); a pose-tracking means (Agrawal teaches at Paragraph [0070] In one embodiment, adjusting the active rear facing camera using the generated new camera settings 266 can include adjusting a directional setting (pose) of the rear facing camera 133A to focus the FOV 362 of rear facing camera 133A on the new second ROI 250B that is determined from the current eye gaze location (i.e., second location 262B). In another embodiment, adjusting the active rear facing camera using the generated new camera settings 266 can include adjusting a zoom level of the rear facing camera 133A to magnify or reduce captured images of the new ROI (i.e., second ROI 250B). In an additional embodiment, adjusting the active rear facing camera using the generated new camera settings 266 can include adjusting a focal distance of the rear facing camera 133A to focus the rear facing camera on the new ROI, i.e., second ROI 250B); at least one camera (Agrawal teaches at Paragraph [0069] that CCM 136 further enables electronic device 100 to generate at least one new camera setting 266 of active rear facing camera 133A at least partially based on the new second ROI 250B. CCM 136 further enables electronic device 100 to adjust the active rear facing camera 133A using the generated new camera setting 266 such that the rear facing camera 133A focuses on the new second ROI 250B. In one embodiment, the generated camera settings 266 can be optimal camera settings 268 that provide an optimal sized and zoomed image on display 130 of the video recording of second ROI 250B); and at least one processor configured to (Agrawal teaches at Paragraph 0031 that CCM 136 includes program code that is executed by processor 102 to enable electronic device 100 to adjust camera settings based on eye gaze tracking and at Paragraph 0049 that CCM 136 enables electronic device 100 to track an eye gaze direction of a user within a display and to automatically adjust an active camera based on the eye gaze tracking):
determine a gaze point and a gaze depth of a user's eyes, by processing gaze-tracking data that is collected by the gaze-tracking means (
Agrawal teaches at Paragraph 0031 that CCM 136 includes program code that is executed by processor 102 to enable electronic device 100 to adjust camera settings based on eye gaze tracking and at Paragraph 0049 that CCM 136 enables electronic device 100 to track an eye gaze direction of a user within a display and to automatically adjust an active camera based on the eye gaze tracking.
Agrawal teaches at Paragraph 0055 that eye gaze location 262 is a specific location that a user of electronic device 100 is looking, which correlates to and is determined based on the eye gaze direction 260. Eye gaze location 262 can include a plurality of discrete locations, such as first location 262A, second location 262B, and third location 262C. According to one embodiment, eye gaze location 262 can include the orientation to the location where the user is looking based on the corresponding eye gaze direction. For example, eye gaze location 262 can have a vertical orientation of 5 degrees upwards, and a horizontal direction that is positive (right) 15 degrees from the front surface 176 of electronic device 100. It is appreciated that the dimensions and orientations measurements would be based on the front surface of electronic device 100 representing a vertical/Y plane, in a 3-dimensional X-Y-Z axes coordinate plane extending through a center point of the front surface of electronic device 100.
Agrawal teaches at Paragraph [0069] that, referring to FIG. 3E, electronic device 100 is further shown after electronic device 100 has (i) determined, based on detection of second eye gaze direction 260B towards second location 262B within display 130, that there is a new ROI, (ii) generated at least one new camera setting and (iii) adjusted at least one camera setting of the active rear facing camera. Specifically, CCM 136 enables electronic device 100 to map second location 262B (i.e., grid 139S) on display 130 to a new second ROI 250B containing dog 326. CCM 136 further enables electronic device 100 to generate at least one new camera setting 266 of active rear facing camera 133A at least partially based on the new second ROI 250B. CCM 136 further enables electronic device 100 to adjust the active rear facing camera 133A using the generated new camera setting 266 such that the rear facing camera 133A focuses on the new second ROI 250B. In one embodiment, the generated camera settings 266 can be optimal camera settings 268 that provide an optimal sized and zoomed image on display 130 of the video recording of second ROI 250B.
Agrawal teaches at Paragraph [0070] In one embodiment, adjusting the active rear facing camera using the generated new camera settings 266 can include adjusting a directional setting (pose) of the rear facing camera 133A to focus the FOV 362 of rear facing camera 133A on the new second ROI 250B that is determined from the current eye gaze location (i.e., second location 262B). In another embodiment, adjusting the active rear facing camera using the generated new camera settings 266 can include adjusting a zoom level of the rear facing camera 133A to magnify or reduce captured images of the new ROI (i.e., second ROI 250B). In an additional embodiment, adjusting the active rear facing camera using the generated new camera settings 266 can include adjusting a focal distance of the rear facing camera 133A to focus the rear facing camera on the new ROI (i.e., second ROI 250B). In one embodiment, a different one of the rear-facing cameras can be selected to be the active rear camera in instances in which the previous active rear camera is unable to provide a good or complete view of the ROI within the display. For example, when the ROI is at a periphery of the FOV of a standard rear camera, a wide angled camera can be selected to be the active rear camera to be able to focus on the entire ROI without having to crop the edges of the video images. Other camera settings adjustments can be made based on the detected amount of light, exposure settings and other identified image characteristics of the new ROI);
control the at least one camera to capture a real-world image of a real-world environment, by adjusting camera settings according to the gaze point and the gaze depth (
Agrawal teaches at Paragraph 0031 that CCM 136 includes program code that is executed by processor 102 to enable electronic device 100 to adjust camera settings based on eye gaze tracking and at Paragraph 0049 that CCM 136 enables electronic device 100 to track an eye gaze direction of a user within a display and to automatically adjust an active camera based on the eye gaze tracking.
Agrawal teaches at Paragraph 0055 that eye gaze location 262 is a specific location that a user of electronic device 100 is looking, which correlates to and is determined based on the eye gaze direction 260. Eye gaze location 262 can include a plurality of discrete locations, such as first location 262A, second location 262B, and third location 262C. According to one embodiment, eye gaze location 262 can include the orientation to the location where the user is looking based on the corresponding eye gaze direction. For example, eye gaze location 262 can have a vertical orientation of 5 degrees upwards, and a horizontal direction that is positive (right) 15 degrees from the front surface 176 of electronic device 100. It is appreciated that the dimensions and orientations measurements would be based on the front surface of electronic device 100 representing a vertical/Y plane, in a 3-dimensional X-Y-Z axes coordinate plane extending through a center point of the front surface of electronic device 100.
Agrawal teaches at Paragraph [0069] that, referring to FIG. 3E, electronic device 100 is further shown after electronic device 100 has (i) determined, based on detection of second eye gaze direction 260B towards second location 262B within display 130, that there is a new ROI, (ii) generated at least one new camera setting and (iii) adjusted at least one camera setting of the active rear facing camera. Specifically, CCM 136 enables electronic device 100 to map second location 262B (i.e., grid 139S) on display 130 to a new second ROI 250B containing dog 326. CCM 136 further enables electronic device 100 to generate at least one new camera setting 266 of active rear facing camera 133A at least partially based on the new second ROI 250B. CCM 136 further enables electronic device 100 to adjust the active rear facing camera 133A using the generated new camera setting 266 such that the rear facing camera 133A focuses on the new second ROI 250B. In one embodiment, the generated camera settings 266 can be optimal camera settings 268 that provide an optimal sized and zoomed image on display 130 of the video recording of second ROI 250B.
Agrawal teaches at Paragraph [0070] In one embodiment, adjusting the active rear facing camera using the generated new camera settings 266 can include adjusting a directional setting (pose) of the rear facing camera 133A to focus the FOV 362 of rear facing camera 133A on the new second ROI 250B that is determined from the current eye gaze location (i.e., second location 262B). In another embodiment, adjusting the active rear facing camera using the generated new camera settings 266 can include adjusting a zoom level of the rear facing camera 133A to magnify or reduce captured images of the new ROI (i.e., second ROI 250B). In an additional embodiment, adjusting the active rear facing camera using the generated new camera settings 266 can include adjusting a focal distance of the rear facing camera 133A to focus the rear facing camera on the new ROI (i.e., second ROI 250B). In one embodiment, a different one of the rear-facing cameras can be selected to be the active rear camera in instances in which the previous active rear camera is unable to provide a good or complete view of the ROI within the display. For example, when the ROI is at a periphery of the FOV of a standard rear camera, a wide angled camera can be selected to be the active rear camera to be able to focus on the entire ROI without having to crop the edges of the video images. Other camera settings adjustments can be made based on the detected amount of light, exposure settings and other identified image characteristics of the new ROI);
determine a pose of the at least one camera at a time of capturing the real-world image, by processing pose-tracking data that is collected by the pose-tracking means
(Agrawal teaches at Paragraph 0031 that CCM 136 includes program code that is executed by processor 102 to enable electronic device 100 to adjust camera settings based on eye gaze tracking and at Paragraph 0049 that CCM 136 enables electronic device 100 to track an eye gaze direction of a user within a display and to automatically adjust an active camera based on the eye gaze tracking.
Agrawal teaches at Paragraph 0055 that eye gaze location 262 is a specific location that a user of electronic device 100 is looking, which correlates to and is determined based on the eye gaze direction 260. Eye gaze location 262 can include a plurality of discrete locations, such as first location 262A, second location 262B, and third location 262C. According to one embodiment, eye gaze location 262 can include the orientation to the location where the user is looking based on the corresponding eye gaze direction. For example, eye gaze location 262 can have a vertical orientation of 5 degrees upwards, and a horizontal direction that is positive (right) 15 degrees from the front surface 176 of electronic device 100. It is appreciated that the dimensions and orientations measurements would be based on the front surface of electronic device 100 representing a vertical/Y plane, in a 3-dimensional X-Y-Z axes coordinate plane extending through a center point of the front surface of electronic device 100.
Agrawal teaches at Paragraph [0069] that, referring to FIG. 3E, electronic device 100 is further shown after electronic device 100 has (i) determined, based on detection of second eye gaze direction 260B towards second location 262B within display 130, that there is a new ROI, (ii) generated at least one new camera setting and (iii) adjusted at least one camera setting of the active rear facing camera. Specifically, CCM 136 enables electronic device 100 to map second location 262B (i.e., grid 139S) on display 130 to a new second ROI 250B containing dog 326. CCM 136 further enables electronic device 100 to generate at least one new camera setting 266 of active rear facing camera 133A at least partially based on the new second ROI 250B. CCM 136 further enables electronic device 100 to adjust the active rear facing camera 133A using the generated new camera setting 266 such that the rear facing camera 133A focuses on the new second ROI 250B. In one embodiment, the generated camera settings 266 can be optimal camera settings 268 that provide an optimal sized and zoomed image on display 130 of the video recording of second ROI 250B.
Agrawal teaches at Paragraph [0070] In one embodiment, adjusting the active rear facing camera using the generated new camera settings 266 can include adjusting a directional setting (pose) of the rear facing camera 133A to focus the FOV 362 of rear facing camera 133A on the new second ROI 250B that is determined from the current eye gaze location (i.e., second location 262B). In another embodiment, adjusting the active rear facing camera using the generated new camera settings 266 can include adjusting a zoom level of the rear facing camera 133A to magnify or reduce captured images of the new ROI (i.e., second ROI 250B). In an additional embodiment, adjusting the active rear facing camera using the generated new camera settings 266 can include adjusting a focal distance of the rear facing camera 133A to focus the rear facing camera on the new ROI (i.e., second ROI 250B). In one embodiment, a different one of the rear-facing cameras can be selected to be the active rear camera in instances in which the previous active rear camera is unable to provide a good or complete view of the ROI within the display. For example, when the ROI is at a periphery of the FOV of a standard rear camera, a wide angled camera can be selected to be the active rear camera to be able to focus on the entire ROI without having to crop the edges of the video images. Other camera settings adjustments can be made based on the detected amount of light, exposure settings and other identified image characteristics of the new ROI);
identify at least one region of the real-world environment that is represented in the real- world image, based on a spatial geometry of the real-world environment and the pose of the at least one camera (
For example, Agrawal teaches at FIGS. 3A, 3C and 3E that the first ROI 250A, the second ROI 250B and the third ROI 250C are configured with different geometric shapes with change of the edges of the ROI and at Paragraph 0061 that the image characteristics of the ROI are identified wherein the image characteristics include face (geometry) detection and scene (geometry) detection enabling focus on the subject (geometry shape) based on the contrast change of the edges in the ROI. The edges of the ROI represent spatial geometry of the ROI as shown in FIGS. 3A-3E with the different geometry shapes outlining the ROIs.
Agrawal teaches at Paragraph [0061] that image characteristics 272 are attributes identified as associated with an ROI 250 and are used to adjust one or more camera settings to focus the camera on the ROI. Image characteristics 272 can be determined from one or more captured ROI images 254 that have been captured by a rear facing camera. For example, image characteristics 272 can include light levels at the ROI, face detection at the ROI, and scene detection at the ROI. Image characteristics 272 can further include contrast detection autofocus that achieves focus on a subject based on the contrast change of the edges in the ROI. Alternatively, image characteristics 272 can include phase detection autofocus that enables focusing on a subject based on the convergence of two separate beams of light. Image characteristics 272 can further include computer vision (CV), machine learning (ML), and artificial intelligence (AI) based techniques for determining an object of interest within the ROI.
Agrawal teaches at Paragraph [0062] FIG. 3A illustrates electronic device 100 being used by a user 310 to record or capture video data 270. A cat 324 and a dog 326 are located behind rear surface 180 away from electronic device 100. The face 314 of the user 310 includes a pair of eyes 316 that are looking in first eye gaze direction 260A. In FIG. 3A, the user is looking at cat 324 and dog 326 along first eye gaze direction 260A towards first ROI 250A. First ROI 250A includes cat 324 and dog 326. Cat 324 and dog 326 are within FOV 346 of the active camera (i.e., rear facing main camera 133A).
Agrawal teaches at Paragraph 0031 that CCM 136 includes program code that is executed by processor 102 to enable electronic device 100 to adjust camera settings based on eye gaze tracking and at Paragraph 0049 that CCM 136 enables electronic device 100 to track an eye gaze direction of a user within a display and to automatically adjust an active camera based on the eye gaze tracking.
Agrawal teaches at Paragraph 0055 that eye gaze location 262 is a specific location that a user of electronic device 100 is looking, which correlates to and is determined based on the eye gaze direction 260. Eye gaze location 262 can include a plurality of discrete locations, such as first location 262A, second location 262B, and third location 262C. According to one embodiment, eye gaze location 262 can include the orientation to the location where the user is looking based on the corresponding eye gaze direction. For example, eye gaze location 262 can have a vertical orientation of 5 degrees upwards, and a horizontal direction that is positive (right) 15 degrees from the front surface 176 of electronic device 100. It is appreciated that the dimensions and orientations measurements would be based on the front surface of electronic device 100 representing a vertical/Y plane, in a 3-dimensional X-Y-Z axes coordinate plane extending through a center point of the front surface of electronic device 100.
Agrawal teaches at Paragraph [0069] that, referring to FIG. 3E, electronic device 100 is further shown after electronic device 100 has (i) determined, based on detection of second eye gaze direction 260B towards second location 262B within display 130, that there is a new ROI, (ii) generated at least one new camera setting and (iii) adjusted at least one camera setting of the active rear facing camera. Specifically, CCM 136 enables electronic device 100 to map second location 262B (i.e., grid 139S) on display 130 to a new second ROI 250B containing dog 326. CCM 136 further enables electronic device 100 to generate at least one new camera setting 266 of active rear facing camera 133A at least partially based on the new second ROI 250B. CCM 136 further enables electronic device 100 to adjust the active rear facing camera 133A using the generated new camera setting 266 such that the rear facing camera 133A focuses on the new second ROI 250B. In one embodiment, the generated camera settings 266 can be optimal camera settings 268 that provide an optimal sized and zoomed image on display 130 of the video recording of second ROI 250B.
Agrawal teaches at Paragraph [0070] In one embodiment, adjusting the active rear facing camera using the generated new camera settings 266 can include adjusting a directional setting (pose) of the rear facing camera 133A to focus the FOV 362 of rear facing camera 133A on the new second ROI 250B that is determined from the current eye gaze location (i.e., second location 262B). In another embodiment, adjusting the active rear facing camera using the generated new camera settings 266 can include adjusting a zoom level of the rear facing camera 133A to magnify or reduce captured images of the new ROI (i.e., second ROI 250B). In an additional embodiment, adjusting the active rear facing camera using the generated new camera settings 266 can include adjusting a focal distance of the rear facing camera 133A to focus the rear facing camera on the new ROI (i.e., second ROI 250B). In one embodiment, a different one of the rear-facing cameras can be selected to be the active rear camera in instances in which the previous active rear camera is unable to provide a good or complete view of the ROI within the display. For example, when the ROI is at a periphery of the FOV of a standard rear camera, a wide angled camera can be selected to be the active rear camera to be able to focus on the entire ROI without having to crop the edges of the video images. Other camera settings adjustments can be made based on the detected amount of light, exposure settings and other identified image characteristics of the new ROI).
Agrawal does not explicitly teach the claim limitation:
a display apparatus comprising: a gaze-tracking means; a pose-tracking means; at least one camera; and at least one processor configured to:
determine whether a representation of the at least one region in at least one of: the real- world image, a previously-captured real-world image, satisfies a quality criteria, wherein the previously-captured image is stored at a data repository that is communicably coupled with the at least one processor;
when it is determined that the representation of the at least one region in the at least one of: the real-world image, the previously-captured real-world image, fails to satisfy the quality criteria, control the at least one camera to capture a reference real-world image representing the at least one region, by adjusting the camera settings such that said representation fulfills the quality criteria, while maintaining the pose of the at least one camera at the time of capturing the real-world image with the representation;
generate training data comprising reference data and input data, wherein the reference data comprises the reference real-world image representing the at least one region, and the input data comprises the at least one of: the real- world image of the at least one region, the previously-captured real-world image of the at least one region; and
send the training data to a processor that is configured to train a first neural network to generate real-world images that satisfy the quality criteria by processing real-world images that fail to satisfy the quality criteria.
However, Malin teaches the new claim limitation: maintaining the pose of the at least one camera at the time of capturing the real-world image with the representation (
Malin teaches at Paragraph 0020 and Paragraph 0041 that the calibration parameters include the focal length, decentering of the optical elements, distortion parameters other than the rotation and displacement parameters for changing the quality of the image (see Paragraph 0043.
It would have been obvious to one of the ordinary skill in the art before the filing date of the instant application to have incorporated Malin’s improvement of the image quality using the camera parameters other than the pose of the camera to achieve quality improvement to have adjusted capture parameters for subsequent frames to achieve image capture that falls within acceptable quality criteria. One of the ordinary skill in the art would have been motivated to have provided camera control module for controlling the camera setting to have focused on the region of interest and for controlling the camera setting to have achieved image capture that falls within acceptable quality threshold.
Pizzocchero explicitly teaches the claim limitation: at least one camera; and at least one processor configured to (Pizzocchero teaches at Paragraph 0135 that method steps can be performed by one or more processors executing a computer program to perform functions of the technology by operating on input data and/or generating output data):
determine whether a representation of the at least one region in at least one of: the real- world image, a previously-captured real-world image, satisfies a quality criteria, wherein the previously-captured image is stored at a data repository that is communicably coupled with the at least one processor (
Pizzocchero teaches at Paragraph [0059] that as document detection and tracking module 105a tracks the physical document throughout the one or more images, module 105a also assesses imaging conditions in the images in order to dynamically adjust (step 202b) one or more operational parameters of image capture device 103 based upon one or more imaging conditions associated with the physical document, as detected in one or more images of the sequence of images. In some embodiments, document detection and tracking module 105a compares imaging conditions such as lighting characteristics of the background in the image with lighting characteristics of the document and adjusts operational parameters of image capture device 103 based upon the comparison. For example, if the background of the image is very bright and the document is dark relative to the background, document detection and tracking module 105a can adjust exposure settings of image capture device 103 to ensure that the maximum possible image signal is acquired from the document. A variety of different approaches can be used by document detection and tracking module 105a to adjust operational parameters of image capture device 103, such as 1) a rule-based approach (e.g., if background and/or document brightness falls within a range of values and/or a threshold value, adjust exposure settings accordingly to maximize signal from the document); 2) a machine learning model trained on a labelled data set; and/or 3) an end-to-end regression model trained on data. Each of these approaches is described in more detail below.
Pizzocchero teaches at Paragraph [0060] Rule-Based Approach: In some embodiments, the rule-based approach leverages heuristics to define capture settings of image capture device 103 given a set of assessed input criteria. An exemplary set of assessed and defined input criteria are as follows: [0061] If ambient light is too bright, module 105a can adjust image capture device 103 parameters to reduce exposure setting and gain; Pizzocchero teaches at Paragraph [0062] If conditions are too dark, module 105a can instruct image capture device 103 to capture subsequent/additional frames using increasing flash intensity, and/or increase exposure settings of image capture device 103; [0063] If there is glare present on the document in the frame, module 105a can reduce exposure settings of image capture device 103 and/or reduce gain parameters for image capture device 103);
when it is determined that the representation of the at least one region in the at least one of: the real-world image, the previously-captured real-world image, fails to satisfy the quality criteria, control the at least one camera to capture a reference real-world image representing the at least one region, by adjusting the camera settings such that said representation fulfills the quality criteria, while maintaining the pose of the at least one camera at the time of capturing the real-world image with the representation (
In a non-limiting example, at the time of capturing the real-world image, the capture setting such as the pose of the camera of Pizzocchero is maintained. The capture setting is not associated with the pose of the image capture device 102 without rotation/tilt of the image capture device 102 for a single frame. The capture setting is the flash intensity, flash duration, shutter speed, ISO speed, gain, aperture, light balance, other than the pose of the image capture device 102.
Pizzocchero teaches at Paragraph 0052 that, the physical document and mobile computing device 102 remains stationary during video capture but one or more operational parameters and/or capture settings of image capture device 103 (e.g., flash intensity, flash duration, shutter speed, ISO speed, gain, aperture, light balance, etc.) are modified or adjusted for different frames of the video, in order to cause one or more security features (such as OVDs) on the physical document to become visible or invisible, change color, change appearance, and so forth. In some embodiments, mobile computing device 102 analyzes frames of the video as the frames are being captured and automatically adjusts operational parameters and/or capture settings of image capture device 103 to generate a set of frames with varying imaging conditions, lighting conditions, and/or image characteristics.
Pizzocchero teaches at Paragraph [0064] that Machine Learning Model Trained on Labelled Data Set: In some embodiments, the approach using a machine learning (ML) model trained on a labelled data set moves beyond the simple heuristics of the rule-based approach to utilize deep learning to convert certain lighting characteristics of the incoming frame(s) into multidimensional embeddings and feed the embeddings to a trained classification model executed by module 105a which evaluates the embeddings using weights adjusted for frames taken with known capture settings to determine whether the incoming frame(s) have sufficient lighting parameters or not to be usable for document verification. In this approach, the classification model can determine one or more parameter adjustments for image capture device 103 and module 105a then adjusts capture parameters for subsequent frames to achieve image capture that falls within acceptable lighting conditions.
Pizzocchero teaches at Paragraph [0065] End-to-End Regression Model: In some embodiments, the approach using an end-to-end regression model trained on data enables the most effective control of the scene and capture settings of image capture device 103. Module 105a executes a trained deep learning regression model to perform end-to-end regression of the lighting conditions and capture settings given any scene, and the regression model can optimize for the specifics of the scene so as to maximize the signal acquired from the document and/or OVD while suppressing noise due to visual/optical phenomena.
Pizzocchero teaches at Paragraph [0066] that document detection and tracking module 105a also assesses physical properties of the document in the images in order to adjust operational parameters and/or capture settings of image capture device 103 and document detection and tracking module 105a can adjust operational parameters and/or capture settings of image capture device 103 to ensure that the maximum possible image signal is acquired from the document. For example, document detection and tracking module 105a can be configured to utilize a deep learning classification model that is trained on surface properties of different materials in images to evaluate the incoming frames, classify a likely composition/material of the document depicted in the frames, and adjust operational parameters to adjust capture settings);
generate training data comprising reference data and input data, wherein the reference data comprises the reference real-world image representing the at least one region, and the input data comprises the at least one of: the real- world image of the at least one region, the previously-captured real-world image of the at least one region (
Pizzocchero’s images taken with known capture settings are used as input images (input data) representing the at least document region to adjust the weights of the machine learning model in order to determine the quality metrics of the incoming images (reference data) representing the least document region.
Pizzocchero teaches at Paragraph [0064] that Machine Learning Model Trained on Labelled Data Set: In some embodiments, the approach using a machine learning (ML) model trained on a labelled data set moves beyond the simple heuristics of the rule-based approach to utilize deep learning to convert certain lighting characteristics of the incoming frame(s) (reference frames) into multidimensional embeddings and feed the embeddings to a trained classification model executed by module 105a which evaluates the embeddings using weights adjusted for frames (input data comprising the previously captured real-world image of the document region) taken with known capture settings to determine whether the incoming frame(s) (input data comprising the real-world image data) have sufficient lighting parameters or not to be usable for document verification. In this approach, the classification model can determine one or more parameter adjustments for image capture device 103 and module 105a then adjusts capture parameters for subsequent frames (reference data comprising the real-time captured real-world image) to achieve image capture that falls within acceptable lighting conditions.
Pizzocchero teaches at Paragraph [0066] that document detection and tracking module 105a also assesses physical properties of the document in the images (the input data) in order to adjust operational parameters and/or capture settings of image capture device 103 and document detection and tracking module 105a can adjust operational parameters and/or capture settings of image capture device 103 to ensure that the maximum possible image signal is acquired from the document. For example, document detection and tracking module 105a can be configured to utilize a deep learning classification model that is trained on surface properties of different materials in images (input images) to evaluate the incoming frames (reference frames), classify a likely composition/material of the document depicted in the frames, and adjust operational parameters to adjust capture settings.
); and
send the training data to a processor that is configured to train a first neural network to generate real-world images that satisfy the quality criteria by processing real-world images that fail to satisfy the quality criteria (
Pizzocchero teaches at Paragraph [0064] that Machine Learning Model Trained on Labelled Data Set: In some embodiments, the approach using a machine learning (ML) model trained on a labelled data set moves beyond the simple heuristics of the rule-based approach to utilize deep learning to convert certain lighting characteristics of the incoming frame(s) into multidimensional embeddings and feed the embeddings to a trained classification model executed by module 105a which evaluates the embeddings using weights adjusted for frames taken with known capture settings to determine whether the incoming frame(s) have sufficient lighting parameters or not to be usable for document verification. In this approach, the classification model can determine one or more parameter adjustments for image capture device 103 and module 105a then adjusts capture parameters for subsequent frames to achieve image capture that falls within acceptable lighting conditions).
It would have been obvious to one of the ordinary skill in the art before the filing date of the instant application to have incorporated Pizzocchero’s machine learning model to have modified Agrawal’s camera control module (CCM) for controlling the at one front facing camera and the at least one rear facing camera to have performed one or more parameter adjustments for image capture device 103 and to have allowed the module 105a to adjust capture parameters for subsequent frames to achieve image capture that falls within acceptable lighting conditions according to Pizzocchero’s machine learning model. One of the ordinary skill in the art would have been motivated to have provided camera control module for controlling the camera setting according to Agrawal based on the eye gaze tracking and the camera pose tracking to have focused on the region of interest and for controlling the camera setting according to Pizzochero to achieve image capture that falls within acceptable lighting conditions.
Rao explicitly teaches the claim limitation:
a display apparatus comprising: a gaze-tracking means; a pose-tracking means; at least one camera; and at least one processor configured to:
determine whether a representation of the at least one region in at least one of: the real- world image, a previously-captured real-world image, satisfies a quality criteria, wherein the previously-captured image is stored at a data repository that is communicably coupled with the at least one processor (Rao teaches Page 13, column 2, lines 13-16 that the camera parameter settings are tuned responsive to a perceptual no-reference quality determination indicating a quality of the captured frames being below a quality threshold level);
when it is determined that the representation of the at least one region in the at least one of: the real-world image, the previously-captured real-world image, fails to satisfy the quality criteria, control the at least one camera to capture a reference real-world image representing the at least one region, by adjusting the camera settings such that said representation fulfills the quality criteria, while maintaining the pose of the at least one camera at the time of capturing the real-world image with the representation (
In a non-limiting example, at the time of capturing the video, the pose of the video camera is maintained and camera parameters are not associated with the pose of the image capture device 102 without rotation/tilt of the video camera for a single frame. The capture setting is the brightness, color, contrast and sharpness (see Rao Paragraph 0024) while the camera system 202 of FIG. 2 is in the fixed position (fixed pose).
Rao teaches at Paragraph 0028 that the adaptive perceptual camera tuning (APT) system can utilize a neural network and reinforcement learning (RL) to automatically and adaptively tune camera parameters remotely to generate a high-quality video feed and at Paragraph 0096 that a video/video stream can be acquired using one or more video cameras deployed in various environmental settings in block 502 for use (e.g., real-time use) in video analytics applications (e.g., object/person detection).
Rao teaches Page 13, column 2, lines 13-16 that the camera parameter settings are tuned responsive to a perceptual no-reference quality determination indicating a quality of the captured frames being below a quality threshold level.
Rao teaches at Paragraph 0076 that reinforcement learning using a DNN can be executed in block 220 to automatically adapt the camera parameters to improve the accuracy of AUs in accordance with aspects of the present invention and at Paragraph 0077 that reinforcement learning (RL) can be utilized in block 220 to determine the best camera settings for a particular scene to provide optimal AU accuracy for video analytics tasks. This learning can be performed in an online manner using RL, in which the system 200 can learn the best camera settings in real-time in any of a plurality of environmental conditions.
Rao teaches at Paragraph 0097 that camera parameters (e.g., brightness, color, contrast, and sharpness), can be dynamically and automatically adjusted based on the reward function derived from the perceptual quality estimator.
The camera parameter of Rao is associated with brightness, color, contrast and sharpness and is not associated with the pose of the camera);
generate training data comprising reference data and input data, wherein the reference data comprises the reference real-world image representing the at least one region, and the input data comprises the at least one of: the real- world image of the at least one region, the previously-captured real-world image of the at least one region (
Rao teaches at Paragraph 0097 that camera parameters (e.g., brightness, color, contrast, and sharpness), can be dynamically and automatically adjusted based on the reward function derived from the perceptual quality estimator.
Rao teaches at Paragraph 0076 that reinforcement learning using a DNN can be executed in block 220 to automatically adapt the camera parameters to improve the accuracy of AUs in accordance with aspects of the present invention and at Paragraph 0077 that reinforcement learning (RL) can be utilized in block 220 to determine the best camera settings for a particular scene to provide optimal AU accuracy for video analytics tasks. This learning can be performed in an online manner using RL, in which the system 200 can learn the best camera settings in real-time in any of a plurality of environmental conditions.
Rao teaches at Paragraph 0028 that the adaptive perceptual camera tuning (APT) system can utilize a neural network and reinforcement learning (RL) to automatically and adaptively tune camera parameters remotely to generate a high-quality video feed and at Paragraph 0096 that a video/video stream can be acquired using one or more video cameras deployed in various environmental settings in block 502 for use (e.g., real-time use) in video analytics applications (e.g., object/person detection). A sequence of video frames captured using a camera can be monitored, captured, and/or received (e.g., by a server 206 with reference to FIG. 2) in block 502. In some embodiments, at 504, the method can include dynamically determining optimal camera parameters to capture superior-quality video frames in response to changing environmental conditions and scene context using Reinforcement Learning (RL) and a Convolutional Neural Network (CNN) in accordance with aspects of the present invention. At 506, the method can include processing the captured video frames with an analytics unit to generate insights, including identifying specific objects, tracking behaviors, and detecting anomalies using RL. In block 508, the RL can be integrated with a perceptual quality estimator (as a reward function) based on a convolutional neural network (CNN), where the estimator can assess video frame quality based on human perceptual aspects); and
send the training data to a processor that is configured to train a first neural network to generate real-world images that satisfy the quality criteria by processing real-world images that fail to satisfy the quality criteria (
Rao teaches at Paragraph 0097 that camera parameters (e.g., brightness, color, contrast, and sharpness), can be dynamically and automatically adjusted based on the reward function derived from the perceptual quality estimator. Rao teaches at Paragraph 0096 that a video/video stream can be acquired using one or more video cameras deployed in various environmental settings in block 502 for use (e.g., real-time use) in video analytics applications (e.g., object/person detection). A sequence of video frames captured using a camera can be monitored, captured, and/or received (e.g., by a server 206 with reference to FIG. 2) in block 502. In some embodiments, at 504, the method can include dynamically determining optimal camera parameters to capture superior-quality video frames in response to changing environmental conditions and scene context using Reinforcement Learning (RL) and a Convolutional Neural Network (CNN) in accordance with aspects of the present invention. At 506, the method can include processing the captured video frames with an analytics unit to generate insights, including identifying specific objects, tracking behaviors, and detecting anomalies using RL. In block 508, the RL can be integrated with a perceptual quality estimator (as a reward function) based on a convolutional neural network (CNN), where the estimator can assess video frame quality based on human perceptual aspects.
Rao teaches Page 13, column 2, lines 13-16 that the camera parameter settings are tuned responsive to a perceptual no-reference quality determination indicating a quality of the captured frames being below a quality threshold level).
It would have been obvious to one of the ordinary skill in the art before the filing date of the instant application to have incorporated Rao’s CNN to have modified Agrawal’s camera control module (CCM) for controlling the at one front facing camera and the at least one rear facing camera to have tuned the camera parameter settings responsive to a perceptual no-reference quality determination indicating a quality of the captured frames being below a quality threshold level according to Rao’s CNN. One of the ordinary skill in the art would have been motivated to have provided camera control module for controlling the camera setting based on the eye gaze tracking and the camera pose tracking to have focused on the region of interest according to Agrawal and for controlling the camera setting according to Rao responsive to a perceptual no-reference quality determination indicating a quality of the captured frames being below a quality threshold level.
Grant explicitly teaches the claim limitation:
determining whether a representation of the at least one region in at least one of: the real- world image, a previously-captured real-world image, satisfies a quality criteria, wherein the previously-captured image is stored at a data repository (Grant teaches at Paragraph 0058 that the image analysis application determines resolution adjustment patterns among respective discrete objects in one or more previously captured images by applying at least one classification machine learning algorithm and/or at least one clustering machine learning algorithm. The image analysis application determines type and/or degree of resolution adjustment (e.g., type and/or degree of a blurring effect) among respective discrete objects by applying a classification algorithm. Additionally or alternatively, the image analysis application determines level of resolution adjustment among respective discrete objects by applying a quality threshold (QT) clustering algorithm);
when it is determined that the representation of the at least one region in the at least one of: the real-world image, the previously-captured real-world image, fails to satisfy the quality criteria, controlling the at least one camera for capturing a reference real-world image representing the at least one region, by adjusting the camera settings such that said representation fulfills the quality criteria, while maintaining the pose of the at least one camera at the time of capturing the real-world image with the representation (
In other words, increasing resolution or increasing focus in Grant involves changing the zoom level or changing the focal length (while maintaining the pose of the image capture device) and does not involve changing the image capture parameter such as the pose of the image capture device.
Grant teaches at Paragraph 0058 that the image analysis application determines resolution adjustment patterns among respective discrete objects in one or more previously captured images by applying at least one classification machine learning algorithm and/or at least one clustering machine learning algorithm. The image analysis application determines type and/or degree of resolution adjustment (e.g., type and/or degree of a blurring effect) among respective discrete objects by applying a classification algorithm. Additionally or alternatively, the image analysis application determines level of resolution adjustment among respective discrete objects by applying a quality threshold (QT) clustering algorithm);
generate training data comprising reference data and input data, wherein the reference data comprises the reference real-world image representing the at least one region, and the input data comprises the at least one of: the real- world image of the at least one region, the previously-captured real-world image of the at least one region (Grant teaches at Paragraph 0058 that the image analysis application applies at least one machine learning algorithm based upon the parsed metadata derived from the archived image capture data and the collected profile data associated with the client and at Paragraph 0061 that image analysis application identifies any such representation based upon audiovisual processing of one or more of the plurality of contextual inputs, e.g., processing of visual and/or audio aspects associated with one or more recently captured images.
Grant teaches at Paragraph 0058 that the image analysis application determines contextual details associated with the archived image capture data and/or the collected profile data associated with the client by applying at least one audiovisual processing machine learning algorithm to inputs derived from parsed image metadata (e.g., associated with one or more previously captured images) and/or parsed audiovisual metadata. According to such further embodiment, the image analysis application identifies respective discrete objects in one or more previously captured images by applying at least one object detection algorithm. One object detection algorithm option is a R-CNN algorithm. The image analysis application optionally determines whether a respective discrete object in one or more previously captured images is visible (e.g., in terms of resolution) based upon application of a R-CNN algorithm in conjunction with analysis of any Boolean metadata pertaining to the respective discrete object. Another object detection algorithm option is a YOLO algorithm. The image analysis application determines resolution adjustment patterns among respective discrete objects in one or more previously captured images by applying at least one classification machine learning algorithm and/or at least one clustering machine learning algorithm. The image analysis application determines type and/or degree of resolution adjustment (e.g., type and/or degree of a blurring effect) among respective discrete objects by applying a classification algorithm. Additionally or alternatively, the image analysis application determines level of resolution adjustment among respective discrete objects by applying a quality threshold (QT) clustering algorithm); and
sending the training data to a processor that is configured to train a first neural network for generating real-world images that satisfy the quality criteria by processing real-world images that fail to satisfy the quality criteria (Grant teaches at Paragraph 0058 that the image analysis application determines resolution adjustment patterns among respective discrete objects in one or more previously captured images by applying at least one classification machine learning algorithm and/or at least one clustering machine learning algorithm. The image analysis application determines type and/or degree of resolution adjustment (e.g., type and/or degree of a blurring effect) among respective discrete objects by applying a classification algorithm. Additionally or alternatively, the image analysis application determines level of resolution adjustment among respective discrete objects by applying a quality threshold (QT) clustering algorithm.
Grant teaches at Paragraph 0065 that the emphasis of one discrete object relative to the others may entail emphasizing one or more photographic parameters of such discrete object (e.g., increasing resolution, increasing focus, increasing contrast, etc.). Per step 230, the image analysis application presents to Client A the captured image modification options, e.g., via a viewfinder of the image capture device. Accordingly, Client A may select one or more of the captured image modification options, depending upon whether Client A prefers emphasis of the oak tree, emphasis of the mountain, or (optionally) emphasis of both the oak tree and the mountain in the at least one captured image.
Grant teaches at Paragraph [0066] that, per step 235 the image analysis application may facilitate at least one further photographic adjustment to the at least one captured image in the context of the example scenario, e.g., based upon at least one comment received from Client A and/or based upon data collected by at least one monitoring sensor associated with the image capture device of Client A. Optionally, per step 240, the image analysis application may facilitate at least one adjustment for subsequent image capture. For instance, the image analysis application may facilitate adjustment of at least one pre-capture image parameter in view of the image capture learning model and/or may facilitate adjustment of at least one element of photographic equipment (e.g., a gimbal) in view of the image capture learning model prior to capture of a subsequent image via the image capture device of Client A.
Grant teaches at Paragraph 0046 that the image analysis application provides the client an option to adjust post-capture image parameters with respect to the at least one captured image in view of the image capture learning model. In the context of the various embodiments, post-capture image parameters pertain to digital image adjustments made following capture.
Grant teaches at Paragraph 0051 that the image analysis application facilitates adjustment of at least one element of photographic equipment at step 240 in view of the image capture learning model prior to capture of a subsequent image via the image capture device. According to such further embodiment, the image analysis application optionally facilitates adjustment of a gimbal and/or other photographic stabilization component attached to or otherwise associated with the image capture device. The image analysis application optionally relays at least one control signal to the gimbal and/or other photographic stabilization component in order to facilitate such adjustment. Additionally or alternatively, based upon the image capture learning model, the image analysis application optionally facilitates execution of autonomous or semi-autonomous adjustment of photographic equipment for subsequent image capture. Additionally or alternatively, based upon the image capture learning model, the image analysis application provides to the client at least one photographic equipment adjustment instruction, e.g., via the client interface of the image capture device and/or via another client communication channel.).
It would have been obvious to one of the ordinary skill in the art before the filing date of the instant application to have incorporated Grant’s machine learning model to have modified Agrawal’s camera control module (CCM) for controlling the at one front facing camera and the at least one rear facing camera to have performed by the image analysis application to facilitate at least one adjustment for subsequent image capture according to Grant’s machine learning model. One of the ordinary skill in the art would have been motivated to have provided camera control module for controlling the camera setting according to Agrawal based on the eye gaze tracking and the camera pose tracking to have focused on the region of interest and for controlling the camera setting according to Grant by the image analysis application to facilitate at least one adjustment for subsequent image capture.
Re Claim 11:
The claim 11 encompasses the same scope of invention as that of the claim 10 except additional claim limitation that when controlling the at least one camera for capturing the reference real-world image, the at least one processor is configured to:
determine reference values of the camera settings based on at least one of: an optical depth of the at least one region, lighting conditions in the at least one region, such that the reference values, when employed, enable the representation of the at least one region in the reference real- world image to satisfy the quality criteria; and
generate a control signal for the at least one camera to employ the determined reference values of the camera settings, for capturing the at least one reference real-world image.
The claim 11 is in parallel with the claim 4 in an apparatus form. The claim 11 is subject to the same rationale of rejection as the claim 4.
Re Claim 12:
The claim 12 encompasses the same scope of invention as that of the claim 10 except additional claim limitation that the at least one processor is configured to: receive, from the processor, weights of the first neural network that are learnt upon the training of the first neural network at the processor; transfer learning of the first neural network to a second neural network, by applying the weights to the second neural network; and process real-world images that do not satisfy the quality criteria, captured by the at least one camera after transferring said learning, to generate corresponding real-world images that satisfy the quality criteria, by employing the second neural network.
The claim 12 is in parallel with the claim 5 in an apparatus form. The claim 12 is subject to the same rationale of rejection as the claim 5.
Re Claim 13:
The claim 13 encompasses the same scope of invention as that of the claim 10 except additional claim limitation that the at least one processor is configured to: generate an extended-reality image using the real-world image; and control at least one display, for displaying the extended-reality image.
The claim 13 is in parallel with the claim 6 in an apparatus form. The claim 13 is subject to the same rationale of rejection as the claim 6.
Re Claim 14:
The claim 14 encompasses the same scope of invention as that of the claim 10 except additional claim limitation that the at least one processor is configured to: generate at least one reprojected real-world image, by time warping the real-world image, during a time period when the at least one camera is controlled for capturing the reference real- world image, wherein upon elapsing of said time period, the at least one processor is configured to control the at least one camera for capturing a next real-world image; generate at least one extended-reality image using the at least one reprojected real-world image; and control at least one display, for displaying the at least one extended-reality image until a next extended-reality image that is generated using the next real-world image is generated for displaying.
The claim 14 is in parallel with the claim 7 in an apparatus form. The claim 14 is subject to the same rationale of rejection as the claim 7.
Re Claim 15:
The claim 15 encompasses the same scope of invention as that of the claim 10 except additional claim limitation that the at least one processor is configured to:
when it is determined that the representation of the at least one region is represented in the at least one of: the real-world image, the previously-captured real-world image, satisfies the quality criteria, control the at least one camera to capture an input real-world image representing the at least one region, by adjusting the camera settings such that said representation fails to satisfy the quality criteria;
generate training data comprising reference data and input data, wherein the reference data comprises the at least one of: the real-world image, the previously-captured real-world image, and the input data comprises the input real-world image; and
send the training data to a processor that is configured to train a first neural network for generating real-world images that satisfy the quality criteria by processing real-world images that fail to satisfy the quality criteria.
The claim 15 is in parallel with the claim 8 in an apparatus form. The claim 15 is subject to the same rationale of rejection as the claim 8.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JIN CHENG WANG whose telephone number is (571)272-7665. The examiner can normally be reached Mon-Fri 8:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, King Poon can be reached at 571-270-0728. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JIN CHENG WANG/Primary Examiner, Art Unit 2617
Prosecution Insights