DETAILED ACTION
Claims 1, 2, 5, 6 and 11-15 are currently pending.
Response to Arguments
Applicant’s arguments with respect to claims 1, 2, 5, 6 and 11-15 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 2, 5, 6 and 11-15 are rejected under 35 U.S.C. 103 as being unpatentable over Watts et al. US Publication 2013/0169844 (hereafter “Watts”) and Khwaja et al. US Publication 2020/0020075 (hereafter “Khwaja”).
Referring to claims 1, 14 and 15, Watts discloses an image processing apparatus comprising:
a first image sensor;
a second image sensor located in a different position from a position of the first image sensor (paragraph 45, In some embodiments, two images may be captured during operation 601 by different cameras or, more specifically, different optical lenses provided on the same device. These images may be referred to as stereo images. In some embodiments, the two cameras are separated by between about 30 millimeters and 150 millimeters);
at least one memory configured to store instructions; and
at least one processor configured to execute the instructions to:
obtain a first image captured by the first image sensor;
obtain a second image captured by the second image sensor (paragraph 45, In some embodiments, two images may be captured during operation 601 by different cameras or, more specifically, different optical lenses provided on the same device);
determine a rough region of interest (RROI) in the first image (paragraph 51, Operation 603 involves detecting the object in each initial image);
determine a geometric transformation that maps a position of an RROI of the second image corresponding to the RROI of the first image to a position of the RROI of the first image (paragraph 52, Operation 604 involves determining an object center line of the object in each initial image as described above with reference to FIGS. 4A and 4B. In some embodiments, other alignment and/or scaling techniques may be used during operation 604);
determine a boundary of a region of interest (ROI) in the first image corresponding to the RROI of the first image, based on the geometric transformation (paragraph 53, In operation 606, the foreground portion may be separated from the background portion);
obtain a third image by applying the geometric transformation to the second image to map the RROI of the second image onto an image plane of the first image (paragraph 52, The method continues with cross-fading the two initial images along the object center line thereby yielding a combined image during operation 605);
determine a static region between the first image and the third image (paragraph 50, The stereo disparity may be used during detecting operation 602 to determine proximity of each pixel or patch in the stereo images to the camera and therefore to identify at least the background portion of the image); and
determine the boundary of the ROI of the first image based on the static region, wherein the static region includes pixels in the first image (paragraph 50, The stereo disparity may be used during detecting operation 602 to determine proximity of each pixel or patch in the stereo images to the camera and therefore to identify at least the background portion of the image) (paragraph 47, In general, a parallax is a displacement or difference in the apparent position of an object viewed along two different lines of sight. It may be represented by the angle or semi-angle of inclination between those two lines. Nearby objects have a larger parallax than more distant objects when observed from different positions, which allows using the parallax values to determine distances and separate foreground and background portions of an image).
Watts does not disclose expressly using a gradient descent.
Khwaja discloses wherein the at least one processor is further configured to execute the instructions to:
determine a parameter of the geometric transformation using a gradient descent that decreases an error between a position of the feature point of the RROI of the first image and a transformed position of the feature point of the RROI of the second image according to the geometric transformation (paragraph 105, In particular embodiments, images (e.g., I-6 and I-7) may be stitched together by performing a global optimization using the results of the heuristic optimization, initial conditions, optimization costs, or any combination thereof. These factors may be globally optimized using, for example gradient descent),
wherein the gradient descent is applied by altering a zoom level that is a transformation parameter of the geometric transformation applied to the second image (paragraph 105, In particular embodiments, the system of parameters may be global optimized through a system of equations that includes a scaling of the image).
At the time of the effective filing date of the claimed invention, it would have obvious to a person of ordinary skill in the art to use a parameter of the geometric transformation using a gradient descent. The motivation for doing so would have been to increase the accuracy of aligning images. Therefore, it would have been obvious to combine Khwaja with Watts to obtain the invention as specified in claims 1, 14 and 15.
Referring to claim 2, Watts discloses wherein the geometric transformation includes at least one of translation, scaling, or perspective change (paragraph 52, Operation 604 involves determining an object center line of the object in each initial image as described above with reference to FIGS. 4A and 4B. In some embodiments, other alignment and/or scaling techniques may be used during operation 604).
Referring to claim 5, Watts discloses wherein the parameter of the geometric transformation includes at least one of a translation parameter, a scaling parameter, or a perspective change parameter (paragraph 52, Operation 604 involves determining an object center line of the object in each initial image as described above with reference to FIGS. 4A and 4B. In some embodiments, other alignment and/or scaling techniques may be used during operation 604).
Referring to claim 6, Watts discloses wherein the perspective change parameter is defined based on a relative position of the first image sensor and the second image sensor (paragraph 50, The stereo disparity may be used during detecting operation 602 to determine proximity of each pixel or patch in the stereo images to the camera and therefore to identify at least the background portion of the image).
Referring to claim 11, Watts discloses wherein the at least one processor is further configured to execute the instructions to:
determine a feature point of a background of the first image based on the boundary of the ROI of the first image (paragraph 42, separate object center lines may be identified for different objects, e.g., objects on the foreground and objects on the background);
determine a transformed position of a feature point of a background of the second image corresponding to the feature point of the background of the first image, according to the geometric transformation (paragraph 52, Operation 604 involves determining an object center line of the object in each initial image as described above with reference to FIGS. 4A and 4B. In some embodiments, other alignment and/or scaling techniques may be used during operation 604);
determine a parameter of an image effect for the feature point of the background of the first image, based on a difference between a position of the feature point of the background of the first image and the transformed position of the feature point of the background of the second image (paragraph 9, In some embodiments, the combined background portion is blurred using one or more techniques, such as a circular blurring and a Gaussian blurring. The combined background portion may be blurred adaptively. The adaptive aspect may depend on differences in positions of object center lines on the foreground and background portions. Additional differences in these positions may drive more blurring of the background); and
generate an output image by applying the image effect on the first image, based on the parameter of the image effect (paragraph 9, In some embodiments, the combined background portion is blurred using one or more techniques, such as a circular blurring and a Gaussian blurring. The combined background portion may be blurred adaptively. The adaptive aspect may depend on differences in positions of object center lines on the foreground and background portions. Additional differences in these positions may drive more blurring of the background).
Referring to claim 12, Watts discloses wherein the image effect includes blurring, and the at least one processor is further configured to execute the instructions to:
determine a blur level for the feature point of the background of the first image based on the difference between the position of the feature point of the background of the first image and the transformed position of the feature point of the background of the second image (paragraph 9, In some embodiments, the combined background portion is blurred using one or more techniques, such as a circular blurring and a Gaussian blurring. The combined background portion may be blurred adaptively. The adaptive aspect may depend on differences in positions of object center lines on the foreground and background portions. Additional differences in these positions may drive more blurring of the background).
Referring to claim 13, Watts discloses wherein the image effect includes emulating 3-dimensional (3D) parallax, and the at least one processor is further configured to execute the instructions to:
generate a 3D parallax image based on the first image and the second image by determining a distance between an object corresponding to the ROI of the first image and an object corresponding to the feature point of the background of the first image as higher based on the difference between the position of the feature point of the background of the first image and the transformed position of the feature point of the background of the second image being larger (paragraph 63, However, images captured with cameras 722 and 724 may not be stereo images from which stereo disparity may be determined. Still detection of at least the foreground portion of the stereo images may be performed during operation 726. Various techniques that do not require stereo disparity may be used, such as motion parallax) (paragraph 47, The motion parallax may be used for video images. It is a depth cue that results from a relative motion of objects captured in the image and the capturing device. In general, a parallax is a displacement or difference in the apparent position of an object viewed along two different lines of sight. It may be represented by the angle or semi-angle of inclination between those two lines. Nearby objects have a larger parallax than more distant objects when observed from different positions, which allows using the parallax values to determine distances and separate foreground and background portions of an image [showing that in applying motion parallax to the combined image requires determining distances between the object and background]).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PETER K HUNTSINGER whose telephone number is (571)272-7435. The examiner can normally be reached Monday - Friday 8:30 - 5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Benny Q Tieu can be reached at 571-272-7490. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PETER K HUNTSINGER/Primary Examiner, Art Unit 2682
