Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
Claims 1-17 are still pending for consideration.
Response to Arguments
Applicants’ arguments filed on May 20, 2026, have been fully considered but they are not persuasive.
Regarding claim 1, Mizukura teaches a computer-implemented image-processing method including obtaining a sequence of captured images while changing an in-focus position over time, thereby teaching capture of images at different focusing positions. Mizukura further teaches performing depth-of-filed extension processing based on distance information and point-spread-function (PSF) information. In particular, Mizukura teaches the amount of blur corresponding to a depth position may be determined and that “reverse PSF filtering processing (deconvolution)” may be performed “to a blurred portion in a captured image” to obtain an EDoF image in which the blur has been removed (see para [0084]-[0085], and [0091]).
Mizukura does not expressly teach determining the applicable PSF based on correlation between corresponding pixels of differently focused first and second images.
Ogura, however, teaches a depth-from-defocus (DFD) technique using two mages having different defocusing amounts intentionally produced by changing focal positions. Ogura explains that the DFD method utilizes “correlation values of defocusing amounts” and that object distance is measured based on PSF and the difference in defocusing amounts. Ogura further teaches that DFD processor 161 convolves plural PSFs with observed pixels SA to produce plural observed pixels CA, compares those pixels with reference pixel SB at corresponding coordinates, and selects the observed pixel having the smallest difference from the reference pixel. Ogura thereafter determines the object distance corresponding to the PSF associated with that comparison result (see para [0016], [0032], and [0036-[0038]).
It would have been obvious to one of ordinary skills in the art to employ Ogura’s correlation-based identification of applicable PSF in Mizukura’s EDoF processing because Mizukura expressly performs EDoF correction based upon PSF information and distance information. Ogura provides a known technique for identifying PSF corresponding to the observed defocus condition from differently focused captured images. Using the PSF identified according to Ogura as the PSF information supplied to Mizukura’s EDoF processing would predictably provide a blur model corresponding to the observed optical condition and thereby improve the accuracy of the reverse-PSF filtering used to remove blur.
On page 1 of the “Remarks” Applicant asserts “Applicant respectfully submits that the combination of Mizukura and Ogura fails to disclose or suggest: determining a point spread function for the at least one camera, based on a correlation between pixels of at least the part of the first image and respective pixels of the corresponding part of the second image, and a first focusing distance range covered by a depth of field of the at least one camera around the first focusing distance, as recited by claim 1”. Applicant further argue that Ogura does not determine a PSF because its candidate PSFs are previously stored and the DFD processor ultimately determines object distance.
Response:
Examiner respectfully disagrees with applicant’s argument. Ogura teaches determining which PSF corresponds to the observed defocus condition by applying candidate PSFs to pixels of one differently focused image and comparing the resulting pixels with respective pixels of the other differently focused image (see para [0036]- [0037]). Although the candidate PSFs nay be stored beforehand, claim 1 does not require dynamically generating, mathematically deriving, or creating a previously known PSF. Rather claim 1 requires “determining a point spread function…..based on correlation pixels”. Ogura determines which candidate PSF corresponds to the observed defocus relationship by convolving respective candidates PSFs with pixels of the one image, comparing the resulting pixels with respective corresponding pixels of another differently focused image and identifying the result having the smallest difference. The fact that Ogura thereafter uses the identified PSF to determine object distance does not negate the antecedent identification of the applicable PSF from the image comparison.
Applicant additionally on page 2 asserts “Furthermore, the combination of Mizukura and Ogura fails to disclose, suggest, or teach the claimed three-image pipeline where a first image and a second image are captured consecutively, a point spread function is determined, and an extended depth of field correction is applied to at least one segment of a third image”.
Response:
Mizukura on para [0090]-[0091] teaches repeated frame-sequential capture while changing the in-focus position and teaches applying PSF-based EDoF correction to blurred portions of captured images. in the modified, system, after the application PSF has been identified from differently focused capture images according to Ogura, it would have been obvious to use that identified PSF as the PSF information in Mizukura’s disclosed EDoF processing of another captured image having a corresponding out-of-focus condition. Such use constitutes applying the known PSF information determined for the camera to another image in the continuing focus-having image sequence according to its established function as a blur model for deconvolution.
Applicant further argues that Mizukara uses only pre-acquired PSF and therefore would not use the PSF identified according to Ogura. This argument addresses the references individually rather than the proposed combination. Mizukara expressly performs EDoF processing based on both distance information and PSF information and use reverse PSF filtering to remove blur from a blurred portion of a captured image. Ogura teaches a technique for identifying the PSF corresponding to an observed defocus condition from differently focused images. It would have been obvious to use Ogura’s identified PSDF as the PSF information employed in Mizukura’s EDoF processing so that the deconvolution filter corresponds to the observed blur condition.
Applicant characterization of claim 1 as requiring a “dynamically generated” PSF or a mathematical deviation of a new blur kernel is also not persuasive because those requirements are not cited in the claim. Thus, Applicant’s arguments are not persuasive.
The rest of pending dependent claims stand or fall with the arguments presented for
independent claim 1, and 9 respectively.
For the reasons stated above, the rejections of claims 1-17 are maintained.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
Claims 1-3, 5-6, 8-11, 13-14, 16-17 are rejected under 35 U.S.C. 103 as being unpatentable over Mizukura et al. (US 20190158803 A1) in view of Ogura (US 20150281554 A1) for the same reason as stated in the last Office Action, the ground is repeated below.
Regarding claim 1, Mizukura et al. teaches a computer-implemented method comprising: obtaining at least one sequence of images of a real-world environment captured consecutively using at least one camera (see para [0025]; “FIG. 11 is a schematic illustration showing an example of preparing two sheets of images having been photographed by shifting in-focus positions by photographing by shifting time around each frame so as to change an in-focus position” Note: a camera at shifted focus distances captures multiple stereo images, so the “real-world environment” language met), wherein an optical focus of the at least one camera is switched between different focussing distances whilst capturing consecutive images of said sequence for a given pair of a first image and a second image that are captured consecutively in said sequence by adjusting the optical focus of the at least one camera at a first focussing distance and a second focussing distance, respectively (see para [0028]; “performing image processing to make a state such that a difference between left and right images becomes as small as possible except a difference in blur”, see also [0074]; “a right image A and a left image B that are different in focusing position, in Step S10 of FIG. 2. As shown in FIG. 3, in the right image A, a photographic subject a at the center is in-focus, and a photographic subject b at the periphery (background) is blurred. On the other hand, in the right image B, a photographic subject a at the center is blurred, and a photographic subject b at the periphery is in-focus”, and para [0103]; “in the vicinity of a central photographic subject, the image information of the right image A is almost used, and with regard to the background, the image information of the left image B is used” Note: distinguishes sharp and blurred regions across paired images), and for a third image of the real-world environment captured by adjusting the optical focus of the at least one camera at a third focussing distance (see para [0091]; “as shown in FIG. 11, in the case of photographing by changing in-focus positions A and B while shifting time for each frame, it becomes possible to prepare two sheets of images having been photographed by shifting the in-focus position”, Note; capturing multiple images while switching focus positions, which necessarily includes images beyond the first two, and repeated frame-sequential capture while changing the in-focus position and performing EDoF correction on captured images implies a third image), applying an extended depth-of-field correction to at least one segment of the third image that is out of focus by using the point spread function determined for the at least one camera (see para [0085]; “The depth-of-field extension correction processing section 220 performs the EDoF processing on the basis of distance information and PSF information. At this time, the parallax information and the distance information may be calculated on the camera head 100 side, or may be calculated on the depth-of-field-extension-correction-processing-section 220 side. In the case where the information in FIG. 6 and FIG. 7 has been acquired, the amount of blur corresponding to a depth position can be known. Accordingly, by performing the reverse PSF filtering processing (deconvolution) to a blurred portion in a captured image, it is possible to obtain an EDoF image in which the blur has been removed”, see also para [0010]; “creates a depth-of-field extended image by synthesizing the image for a left eye and the image for a right eye in each of which the depth-of-field has been extended”). However, Mizukura et al. does not teach assuming that at least a part of the first image is in focus, whilst a corresponding part of the second image is out of focus; and determining a point spread function for the at least one camera, based on a correlation between pixels of at least the part of the first image and respective pixels of the corresponding part of the second image and a first focussing distance range covered by a depth of field of the at least one camera around the first focussing distance.
In the same field of endeavor Ogura teaches assuming that at least a part of the first image is in focus, whilst a corresponding part of the second image is out of focus (see para [0032]; “DFD processor 161 is disposed in image processor 160, and performs the DFD calculation to produce a depth map. To be more specific, DFD processor 161 uses two images: observed image PA and reference image PB having different defocusing amounts produced intentionally by changing focal positions”, see also para [0017]; “a defocusing amount is uniquely determined for each image-capturing apparatus in response to a relation between a focal position and the object distance. In the DFD method utilizing the above characteristics, two images having different defocus amounts are produced”, Note: the observed image and reference images are intentionally captured with different defocus amounts produces by changing focal positions. Because the images are captured at different focal positions, corresponding portions of the two images necessaryily exhibit different focus conditions), and determining a point spread function for the at least one camera, based on a correlation between pixels of at least the part of the first image and respective pixels of the corresponding part of the second image (see para [0016]; “A number of methods for measuring an object distance, a distance from an image-capturing apparatus to an object includes a depth from Defocus (DFD) method that utilizes correlation values of defocusing amounts generated in image captured with a camera. In general, a defocusing amount is uniquely determined for each image-capturing apparatus in response to a relation between a focal position and the object distance. In the DFD method utilizing the above characteristics, two images having different defocus amounts are produced, and the object distance is measured based on a point-spread function (PSF) and a difference in the defocusing amounts”, see also para [0036]; “DFD processor 161 produces plural observed pixels CA by convolutions of plural PSFs with observed pixels SA”, and para [0038]; “DFD processor 161 then compares observed pixels CA1 to CA16 with reference pixel SB, and selects observed pixel CAn that has the smallest difference from reference pixel SB among observed pixels CA1 to CA16 …DFD processor 161 determines the object distance corresponding to the point spread function for convolution” Note: determines which PSF corresponds to the observed defocus condition), and a first focussing distance range covered by a depth of field of the at least one camera around the first focussing distance (see para [0032]; “To be more specific, DFD processor 161 uses two images: observed image PA and reference image PB having different defocusing amounts produced intentionally by changing focal positions. DFD processor 161 produces the depth map based on observed image PA, reference image PB, and point spread functions (PSFs). The depth map indicates object distances at respective ones of pixels of observed image PA (reference image PB)”). Accordingly, it would have been obvious to one of ordinary of skill in the art before the effecting filling date of the invention to modify general method of observation by switching between a stereoscopic vision image and a depth-of-field extended image correspondingly to a situation of Mizukura et al. in view of the use of an image-capturing apparatus performs a convenient focusing operation of Ogura in order to improve the correspondence between the deconvolution kernel and the captured image while still performing the same EDoF processing and blur modeling accuracy (see para [0016]).
Regarding claim 2, the rejection of claim 1 is incorporated herein.
Ogura in the combination further teach wherein the step of assuming comprises assuming that an entirety of the first image is in focus, whilst an entirety of the second image is out of focus, and wherein the point spread function is determined based on a correlation between pixels of the first image and respective pixels of the second image, and the first focussing distance range (see para [0016]; “two images having different defocus amounts are produced, and the object distance is measured based on a point-spread function (PSF) and a difference in the defocusing amounts. The image-capturing apparatus in accordance with this embodiment measures the object distance by utilizing the DFD calculation to perform an auto-focus control”, see also para [0032]; “To be more specific, DFD processor 161 uses two images: observed image PA and reference image PB having different defocusing amounts produced intentionally by changing focal positions. DFD processor 161 produces the depth map based on observed image PA, reference image PB, and point spread functions (PSFs). The depth map indicates object distances at respective ones of pixels of observed image PA (reference image PB)”).
Regarding claim 3, the rejection of claim 1 is incorporated herein.
Mizukura et al. in the combination further teach further comprising: obtaining a plurality of depth maps captured corresponding to the images in said sequence (see para [0082]; “In this method, used are spatial distance information (depth map: depth map) as shown in FIG. 6 and the change characteristic, depending on a distance, of a PSF (Point Spread Function) showing how a lens blurs as shown in FIG. 7”, see also para [0084]; “Moreover, FIG. 7 shows a situation that blur occurs correspondingly to a depth relative to an in-focus position, and as a distance from the in-focus position increases, blur becomes large. As shown in FIG. 7, the amount of blur according to a depth can be approximated with a blur function by Pill Box Function and a blur function by a two-dimensional Gauss function”).
Ogura in the combination further teach and identifying at least one image segment of the first image and a corresponding image segment of the second image in which the at least one image segment of the first image is in focus whilst the corresponding image segment of the second image is out of focus, wherein the step of determining the point spread function is performed, based on a correlation between pixels of the at least one image segment of the first image and respective pixels of the corresponding image segment of the second image, and respective optical depths in at least one segment of a first depth map corresponding to the at least one image segment of the first image, wherein said part of the first image comprises the at least one image segment of the first image (see para [0036]; “DFD processor 161 produces plural observed pixels CA by convolutions of plural PSFs with observed pixels SA. DFD processor 161 compares plural observed pixels CA with reference pixels SB located at the same coordinates as pixels CA on the image”, see also para [0032]; “observed image PA and reference image PB having different defocusing amounts produced intentionally by changing focal positions. DFD processor 161 produces the depth map based on observed image PA, reference image PB, and point spread functions (PSFs). The depth map indicates object distances at respective ones of pixels of observed image PA (reference image PB)” Note: It determines depth map and corresponding PSF values at localized pixel regions across the image plane, which necessarily constitutes identifying image segments where the focus condition differs between images).
Regarding claim 5, the rejection of claim 1 is incorporated herein.
Mizukura et al. in the combination further teach further wherein the at least one sequence of images comprises two sequences of images, one of the two sequences comprising left images for a left eye of a user, another of the two sequences comprising right images for a right eye of the user (see Fig. 2, para [0006]; “a photographing situation acquiring section that acquires information with regard to a photographing situation of an image for a left eye or an image for a right eye; and a determining section that determines correspondingly to the information with regard to the photographing situation whether to perform stereoscopic vision image processing for the image for a left eye or the image for a right eye”), the at least one camera comprising a left camera and a right camera (see para [0061]; “it is possible to acquire a distance to an object from an image by utilizing parallax between right and left cameras”), and wherein the step of applying the extended depth-of-field correction comprises applying the extended depth-of-field correction to the left images and the right images in an alternating manner (see para [0080]; “In the depth-of-field extension correction processing section 220, the EDoF processing can be performed independently for each of the right image A and the left image B. In this case, the EDoF is performed with only one sheet of an image for one eye of either left or right. In the EDoF processing using one sheet of an image for one eye, there are a plurality of variations shown in the below”).
Regarding claim 6, the rejection of claim 1 is incorporated herein.
Mizukura et al. in the combination further teach wherein the optical focus of the at least one camera is switched between N different focussing distances whilst capturing consecutive images of said sequence, the N different focussing distances comprising fixed focussing distances (see para [0025]; “FIG. 11 is a schematic illustration showing an example of preparing two sheets of images having been photographed by shifting in-focus positions by photographing by shifting time around each frame so as to change an in-focus position”, see also para [0047]; “The stereoscopic vision/depth-of-field extension switching determining section 210 includes a photographing situation acquiring section 212 that acquires information with regard to photographing situations, such as an optical zoom value, an electronic zoom value, operation information by a user, parallax information, and distance information, and a determining section 214 that determines correspondingly to information with regard to a photographing situation”).
Regarding claim 8, the rejection of claim 1 is incorporated herein.
Mizukura et al. in the combination further teach wherein the third image is any one of: a previous image in said sequence, a subsequent image in said sequence, the second image (see para [0080]; “In the depth-of-field extension correction processing section 220, the EDoF processing can be performed independently for each of the right image A and the left image B. In this case, the EDoF is performed with only one sheet of an image for one eye of either left or right. In the EDoF processing using one sheet of an image for one eye, there are a plurality of variations shown in the below. In this case, since the EDoF is completed by only one eye, the preparation of two sheets of images different in in-focus position becomes a dummy. Accordingly, another one sheet of an image may be made a copy of an image having been subjected to the EDoF”).
Regarding claim 9, the scope of claim 9 is fully encompassed by the scope of claim 1, accordingly, the rejection analysis of claim 1 is equally applicable here (see also para [0047]; “The image processing device 200 includes a stereoscopic vision/depth-of-field extension switching determining section 210, a depth-of-field extension correction processing section 220” of Mizukura et al.).
Regarding claim 10, the rejection of claim 9 is incorporated herein.
Ogura in the combination further teach wherein the at least one server configured to assume that an entirety of the first image is in focus, whilst an entirety of the second image is out of focus, and wherein the point spread function is determined based on a correlation between pixels of the first image and respective pixels of the second image, and the first focussing distance range (see para [0016]; “two images having different defocus amounts are produced, and the object distance is measured based on a point-spread function (PSF) and a difference in the defocusing amounts. The image-capturing apparatus in accordance with this embodiment measures the object distance by utilizing the DFD calculation to perform an auto-focus control”, see also para [0032]; “To be more specific, DFD processor 161 uses two images: observed image PA and reference image PB having different defocusing amounts produced intentionally by changing focal positions. DFD processor 161 produces the depth map based on observed image PA, reference image PB, and point spread functions (PSFs). The depth map indicates object distances at respective ones of pixels of observed image PA (reference image PB)”).
Regarding claim 11, the rejection of claim 9 is incorporated herein.
Mizukura et al. in the combination further teach wherein the at least one server configured to: obtaining a plurality of depth maps captured corresponding to the images in said sequence (see para [0082]; “In this method, used are spatial distance information (depth map: depth map) as shown in FIG. 6 and the change characteristic, depending on a distance, of a PSF (Point Spread Function) showing how a lens blurs as shown in FIG. 7”, see also para [0084]; “Moreover, FIG. 7 shows a situation that blur occurs correspondingly to a depth relative to an in-focus position, and as a distance from the in-focus position increases, blur becomes large. As shown in FIG. 7, the amount of blur according to a depth can be approximated with a blur function by Pill Box Function and a blur function by a two-dimensional Gauss function”).
Ogura in the combination further teach and identifying at least one image segment of the first image and a corresponding image segment of the second image in which the at least one image segment of the first image is in focus whilst the corresponding image segment of the second image is out of focus, wherein the step of determining the point spread function is performed, based on a correlation between pixels of the at least one image segment of the first image and respective pixels of the corresponding image segment of the second image, and respective optical depths in at least one segment of a first depth map corresponding to the at least one image segment of the first image, wherein said part of the first image comprises the at least one image segment of the first image (see para [0036]; “DFD processor 161 produces plural observed pixels CA by convolutions of plural PSFs with observed pixels SA. DFD processor 161 compares plural observed pixels CA with reference pixels SB located at the same coordinates as pixels CA on the image”, see also para [0032]; “observed image PA and reference image PB having different defocusing amounts produced intentionally by changing focal positions. DFD processor 161 produces the depth map based on observed image PA, reference image PB, and point spread functions (PSFs). The depth map indicates object distances at respective ones of pixels of observed image PA (reference image PB)” Note: It determines depth map and corresponding PSF values at localized pixel regions across the image plane, which necessarily constitutes identifying image segments where the focus condition differs between images).
Regarding claim 13, the rejection of claim 9 is incorporated herein.
Mizukura et al. in the combination further teach further wherein the at least one sequence of images comprises two sequences of images, one of the two sequences comprising left images for a left eye of a user, another of the two sequences comprising right images for a right eye of the user (see Fig. 2, para [0006]; “a photographing situation acquiring section that acquires information with regard to a photographing situation of an image for a left eye or an image for a right eye; and a determining section that determines correspondingly to the information with regard to the photographing situation whether to perform stereoscopic vision image processing for the image for a left eye or the image for a right eye”), the at least one camera comprising a left camera and a right camera (see para [0061]; “it is possible to acquire a distance to an object from an image by utilizing parallax between right and left cameras”), and wherein the step of applying the extended depth-of-field correction comprises applying the extended depth-of-field correction to the left images and the right images in an alternating manner (see para [0080]; “In the depth-of-field extension correction processing section 220, the EDoF processing can be performed independently for each of the right image A and the left image B. In this case, the EDoF is performed with only one sheet of an image for one eye of either left or right. In the EDoF processing using one sheet of an image for one eye, there are a plurality of variations shown in the below”).
Regarding claim 14, the rejection of claim 9 is incorporated herein.
Mizukura et al. in the combination further teach wherein the optical focus of the at least one camera is switched between N different focussing distances whilst capturing consecutive images of said sequence, the N different focussing distances comprising fixed focussing distances (see para [0025]; “FIG. 11 is a schematic illustration showing an example of preparing two sheets of images having been photographed by shifting in-focus positions by photographing by shifting time around each frame so as to change an in-focus position”, see also para [0047]; “The stereoscopic vision/depth-of-field extension switching determining section 210 includes a photographing situation acquiring section 212 that acquires information with regard to photographing situations, such as an optical zoom value, an electronic zoom value, operation information by a user, parallax information, and distance information, and a determining section 214 that determines correspondingly to information with regard to a photographing situation”).
Regarding claim 16, the rejection of claim 9 is incorporated herein.
Mizukura et al. in the combination further teach wherein the third image is any one of: a previous image in said sequence, a subsequent image in said sequence, the second image (see para [0080]; “In the depth-of-field extension correction processing section 220, the EDoF processing can be performed independently for each of the right image A and the left image B. In this case, the EDoF is performed with only one sheet of an image for one eye of either left or right. In the EDoF processing using one sheet of an image for one eye, there are a plurality of variations shown in the below. In this case, since the EDoF is completed by only one eye, the preparation of two sheets of images different in in-focus position becomes a dummy. Accordingly, another one sheet of an image may be made a copy of an image having been subjected to the EDoF”).
Regarding claim 17, the rejection of claim 1 is incorporated herein
Mizukura et al. in the combination further teach a computer program product comprising a non-transitory machine- readable data storage medium having stored thereon program instructions that, when executed by a processor, cause the processor to execute steps of a computer-implemented method (see Fig. 1, para [0023]-[0029]; “Image processor 160 processes the image data produced by CMOS image sensor 140 to produce image data to be displayed on monitor display 220 and to produce image data to be stored in memory card 200 …Internal memory 240.. stores a control program that controls entire digital video camera 100. Internal memory 240 also stores point spread functions (PSFs)… An instruction entering through touch panel 220B as a touch action is supplied to controller 180 to be processed”).
Claims 4, 7, 12, and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Mizukura et al. in view of Ogura as applied in claim 1, and 9 above and further in view of Ackerman et al. (US 20150003819 A1) for the same reason as stated in the last Office Action, the ground is repeated below.
Regarding claim 4, the rejection of claim 1 is incorporated herein.
Mizukura et al. in the combination further teach and applying the extended depth-of-field correction to the at least one image segment of the third image that is out of focus, only when the at least one image segment of the third image overlaps with the gaze region (see para [0082]; “used are spatial distance information (depth map: depth map) as shown in FIG. 6 and the change characteristic, depending on a distance, of a PSF (Point Spread Function) showing how a lens blurs as shown in FIG. 7”, see also para [0084]; “As shown in FIG. 7, the amount of blur according to a depth can be approximated with a blur function by Pill Box Function and a blur function by a two-dimensional Gauss function” Note: disclose EDoF correction using PSF and depth information). However, the combination of Mizukura et al. and Ogura does not teach further comprising obtaining information indicative of a gaze direction of a user, determining a gaze region in the third image, based on the gaze direction of the user.
In thew same field of endeavor Ackerman et al. teach further comprising obtaining information indicative of a gaze direction of a user, determining a gaze region in the third image, based on the gaze direction of the user (see para [0005]; “An eye gaze of a user is tracking using an eye tracking system. A vector that corresponds to a direction in which an eye of a user is gazing at a point in time is determined based on the eye tracking. The direction is in a field of view of a camera. A distance is determined based on the vector and a location of a lens of the camera. The lens is automatically focused based on the distance”), (see para [0021]; “FIG. 6 is a flowchart of one embodiment of a process of focusing a camera based on a depth map of locations gazed at by a user” Note: obtaining gaze direction and applying correction (focus adjustment) specifically to the region that overlaps the gaze). Accordingly, it would have been obvious to one of ordinary skill in the art before the effecting filling date of the invention to modify general method of observation by switching between a stereoscopic vision image and a depth-of-field extended image correspondingly to a situation of Mizukura et al. in view of the use of an image-capturing apparatus performs a convenient focusing operation of Ogura and automatically focus a camera based on eye tracking of Ackerman et al. in order for selective correction in gaze region to improve efficiency and user experience (see para [0005]).
Regarding claim 7, the rejection of claim 1 is incorporated herein.
Ackerman et al. in the combination further teach further teach wherein the optical focus of the at least one camera is switched between N different focussing distances whilst capturing consecutive images of said sequence, wherein the N different focussing distances correspond to optical depths at which N users are gazing (see para [0007]; “An eye gaze of a user is tracking using an eye tracking system. A vector that corresponds to a direction in which an eye of a user is gazing at a point in time is determined based on the eye tracking. The direction is in a field of view of a camera. A distance is determined based on the vector and a location of a lens of the camera. The lens is automatically focused based on the distance” Note: teaches dynamically adjusting the optical focus based on gaze position, and by extension, sequentially switching focus between depths when multiple gaze positions are detected). Accordingly, it would have been obvious to one of ordinary skill in the art before the effecting filling date of the invention to modify general method of observation by switching between a stereoscopic vision image and a depth-of-field extended image correspondingly to a situation of Mizukura et al. in view of the use of an image-capturing apparatus performs a convenient focusing operation of Ogura and a technology automatically focus a camera based on eye tracking of Ackerman et al. in order to ensure the captured images correspond to the actual gaze depths of one or more users (see para [0005]).
Regarding claim 12, the rejection of claim 9 is incorporated herein.
Mizukura et al. in the combination further teach and applying the extended depth-of-field correction to the at least one image segment of the third image that is out of focus, only when the at least one image segment of the third image overlaps with the gaze region (see para [0047]; “Moreover, the depth-of-field extension correction processing section 220 includes a captured image acquiring section 22 that acquires an image for a left eye and an image for a right eye, and a depth-of-field extension processing section 224 that extends a depth-of-field for each of an image for a left eye and an image for a right eye and creates a depth-of-field extended image by synthesizing an image for a left eye and an image for a right eye in each of which a depth-of-field has been extended”).
Ackerman et al. in the combination further teach further teach comprising obtaining information indicative of a gaze direction of a user, determining a gaze region in the third image, based on the gaze direction of the user (see para [0005]; “An eye gaze of a user is tracking using an eye tracking system. A vector that corresponds to a direction in which an eye of a user is gazing at a point in time is determined based on the eye tracking. The direction is in a field of view of a camera. A distance is determined based on the vector and a location of a lens of the camera. The lens is automatically focused based on the distance”), see para [0021]; “FIG. 6 is a flowchart of one embodiment of a process of focusing a camera based on a depth map of locations gazed at by a user” Note: obtaining gaze direction and applying correction (focus adjustment) specifically to the region that overlaps the gaze). Accordingly, it would have been obvious to one of ordinary skill in the art before the effecting filling date of the invention to modify general method of observation by switching between a stereoscopic vision image and a depth-of-field extended image correspondingly to a situation of Mizukura et al. in view of the use of an image-capturing apparatus performs a convenient focusing operation of Ogura and a technology automatically focus a camera based on eye tracking of Ackerman et al. in order for selective correction where the user is looking reduces computational load and improves perceived image quality (see para [0005]).
Regarding claim 15, the rejection of claim 9 is incorporated herein.
Ackerman et al. in the combination further teach further teach wherein the optical focus of the at least one camera is switched between N different focussing distances whilst capturing consecutive images of said sequence, wherein the N different focussing distances correspond to optical depths at which N users are gazing (see para [0007]; “An eye gaze of a user is tracking using an eye tracking system. A vector that corresponds to a direction in which an eye of a user is gazing at a point in time is determined based on the eye tracking. The direction is in a field of view of a camera. A distance is determined based on the vector and a location of a lens of the camera. The lens is automatically focused based on the distance” Note: teaches dynamically adjusting the optical focus based on gaze position, and by extension, sequentially switching focus between depths when multiple gaze positions are detected). Accordingly, it would have been obvious to one of ordinary skill in the art before the effecting filling date of the invention to modify general method of observation by switching between a stereoscopic vision image and a depth-of-field extended image correspondingly to a situation of Mizukura et al. in view of the use of an image-capturing apparatus performs a convenient focusing operation of Ogura and a technology automatically focus a camera based on eye tracking of Ackerman et al. in order to ensure the captured images correspond to the actual gaze depths of one or more users (see para [0005]).
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WINTA GEBRESLASSIE whose telephone number is (571)272-3475. The examiner can normally be reached Monday-Friday9:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Bee can be reached at 571-270-5180. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WINTA GEBRESLASSIE/Examiner, Art Unit 2677
Prosecution Insights