DETAILED ACTION
Claims 1-6 and 11-15 are currently pending.
Response to Arguments
Applicant’s arguments with respect to claims 1-6 and 11-15 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 1-6 and 11-15 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claims contain subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
Claim 1 recites “determine a static region between the first image and the third image by calculating an alignment error between a first pixel in the first image and a second pixel in the third image, wherein the second pixel corresponds to the first pixel; and wherein the static region includes pixels in the first image, of which the alignment error is less than a threshold defined in pixel units, the alignment error representing a residual error of the geometric transformation on the image plane.” While the Applicant’s specification discusses determining a static region between the first image and the third image, the specification fails to describe that the static region is determined by “calculating an alignment error between a first pixel in the first image and a second pixel in the third image”. The discussion of a distance error pertains to using a gradient descent that decreases the error, not using the distance error to determine a static region between the first image and the third image. The Applicant’s specification also fails to discuss that “the alignment error is less than a threshold defined in pixel units” or that “the alignment error representing a residual error of the geometric transformation on the image plane”.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3 and 11-15 are rejected under 35 U.S.C. 103 as being unpatentable over Watts et al. US Publication 2013/0169844 as applied to claim 3 above, and further in view of Lin et al. US Publication 2015/0294490 (hereafter “Lin”).
Referring to claims 1, 14 and 15, Watts discloses an image processing apparatus comprising:
a first image sensor;
a second image sensor located in a different position from a position of the first image sensor (paragraph 45, In some embodiments, two images may be captured during operation 601 by different cameras or, more specifically, different optical lenses provided on the same device. These images may be referred to as stereo images. In some embodiments, the two cameras are separated by between about 30 millimeters and 150 millimeters);
at least one memory configured to store instructions; and
at least one processor configured to execute the instructions to:
obtain a first image captured by the first image sensor;
obtain a second image captured by the second image sensor (paragraph 45, In some embodiments, two images may be captured during operation 601 by different cameras or, more specifically, different optical lenses provided on the same device);
determine a rough region of interest (RROI) in the first image (paragraph 51, Operation 603 involves detecting the object in each initial image);
determine a geometric transformation that maps a position of an RROI of the second image corresponding to the RROI of the first image to a position of the RROI of the first image (paragraph 52, Operation 604 involves determining an object center line of the object in each initial image as described above with reference to FIGS. 4A and 4B. In some embodiments, other alignment and/or scaling techniques may be used during operation 604);
determine a boundary of a region of interest (ROI) in the first image corresponding to the RROI of the first image, based on the geometric transformation (paragraph 53, In operation 606, the foreground portion may be separated from the background portion);
obtain a third image by applying the geometric transformation to the second image to map the RROI of the second image onto an image plane of the first image (paragraph 52, The method continues with cross-fading the two initial images along the object center line thereby yielding a combined image during operation 605);
determine a static region between the first image and the third image (paragraph 50, The stereo disparity may be used during detecting operation 602 to determine proximity of each pixel or patch in the stereo images to the camera and therefore to identify at least the background portion of the image); and
determine the boundary of the ROI of the first image based on the static region, wherein the static region includes pixels in the first image, as the ROI of the first image wherein the second pixel corresponds to the first pixel (paragraph 50, The stereo disparity may be used during detecting operation 602 to determine proximity of each pixel or patch in the stereo images to the camera and therefore to identify at least the background portion of the image) (paragraph 47, In general, a parallax is a displacement or difference in the apparent position of an object viewed along two different lines of sight. It may be represented by the angle or semi-angle of inclination between those two lines. Nearby objects have a larger parallax than more distant objects when observed from different positions, which allows using the parallax values to determine distances and separate foreground and background portions of an image).
While Watts discloses aligning the first and second image to determine the boundary of the ROI of the first image, Watts does not disclose expressly that aligning pixels of which a position difference is less than a threshold.
Lin discloses calculating an alignment error between a first pixel in the first image and a second pixel in the third image, wherein the second pixel corresponds to the first pixel (paragraph 50, At selecting operation 424, inliers of matched feature points are selected that fit a model with a distance less than a distance threshold using, for example, RANSAC);
wherein the static region includes pixels in the first image, of which the alignment error is less than a threshold defined in pixel units, the alignment error representing a residual error of the geometric transformation on the image plane (paragraph 52, At aligning operation 428, the first image and the transformed image are aligned using the aligning transformation such that the second image is transformed to the coordinate system to the first image. The aligning operation is to transform the second image using the aligning transformation generated by using inliers of matched feature points).
At the time of the effective filing date of the claimed invention, it would have obvious to a person of ordinary skill in the art to align pixels of which a position difference is less than a threshold.
The motivation for doing so would have been to limit the processing required to map the pictures and to increase the image quality of the combined image. Therefore, it would have been obvious to combine Lin with Watts to obtain the invention as specified in claim 1.
Referring to claim 2, Watts discloses wherein the geometric transformation includes at least one of translation, scaling, or perspective change (paragraph 52, Operation 604 involves determining an object center line of the object in each initial image as described above with reference to FIGS. 4A and 4B. In some embodiments, other alignment and/or scaling techniques may be used during operation 604).
Referring to claim 3, Watts discloses wherein the at least one processor is further configured to execute the instructions to:
determine a feature point of the RROI of the first image;
determine a feature point of the RROI of the second image corresponding to the feature point of the RROI of the first image; and
determine the geometric transformation based on the feature point of the RROI of the first image and the feature point of the RROI of the second image (paragraph 52, Operation 604 involves determining an object center line of the object in each initial image as described above with reference to FIGS. 4A and 4B. In some embodiments, other alignment and/or scaling techniques may be used during operation 604).
Referring to claim 11, Watts discloses wherein the at least one processor is further configured to execute the instructions to:
determine a feature point of a background of the first image based on the boundary of the ROI of the first image (paragraph 42, separate object center lines may be identified for different objects, e.g., objects on the foreground and objects on the background);
determine a transformed position of a feature point of a background of the second image corresponding to the feature point of the background of the first image, according to the geometric transformation (paragraph 52, Operation 604 involves determining an object center line of the object in each initial image as described above with reference to FIGS. 4A and 4B. In some embodiments, other alignment and/or scaling techniques may be used during operation 604);
determine a parameter of an image effect for the feature point of the background of the first image, based on a difference between a position of the feature point of the background of the first image and the transformed position of the feature point of the background of the second image (paragraph 9, In some embodiments, the combined background portion is blurred using one or more techniques, such as a circular blurring and a Gaussian blurring. The combined background portion may be blurred adaptively. The adaptive aspect may depend on differences in positions of object center lines on the foreground and background portions. Additional differences in these positions may drive more blurring of the background); and
generate an output image by applying the image effect on the first image, based on the parameter of the image effect (paragraph 9, In some embodiments, the combined background portion is blurred using one or more techniques, such as a circular blurring and a Gaussian blurring. The combined background portion may be blurred adaptively. The adaptive aspect may depend on differences in positions of object center lines on the foreground and background portions. Additional differences in these positions may drive more blurring of the background).
Referring to claim 13, Watts discloses wherein the image effect includes emulating 3-dimensional (3D) parallax, and the at least one processor is further configured to execute the instructions to:
generate a 3D parallax image based on the first image and the second image by determining a distance between an object corresponding to the ROI of the first image and an object corresponding to the feature point of the background of the first image as higher based on the difference between the position of the feature point of the background of the first image and the transformed position of the feature point of the background of the second image being larger (paragraph 63, However, images captured with cameras 722 and 724 may not be stereo images from which stereo disparity may be determined. Still detection of at least the foreground portion of the stereo images may be performed during operation 726. Various techniques that do not require stereo disparity may be used, such as motion parallax) (paragraph 47, The motion parallax may be used for video images. It is a depth cue that results from a relative motion of objects captured in the image and the capturing device. In general, a parallax is a displacement or difference in the apparent position of an object viewed along two different lines of sight. It may be represented by the angle or semi-angle of inclination between those two lines. Nearby objects have a larger parallax than more distant objects when observed from different positions, which allows using the parallax values to determine distances and separate foreground and background portions of an image [showing that in applying motion parallax to the combined image requires determining distances between the object and background]).
Referring to claim 12, Watts discloses wherein the image effect includes blurring, and the at least one processor is further configured to execute the instructions to:
determine a blur level for the feature point of the background of the first image based on the difference between the position of the feature point of the background of the first image and the transformed position of the feature point of the background of the second image (paragraph 9, In some embodiments, the combined background portion is blurred using one or more techniques, such as a circular blurring and a Gaussian blurring. The combined background portion may be blurred adaptively. The adaptive aspect may depend on differences in positions of object center lines on the foreground and background portions. Additional differences in these positions may drive more blurring of the background).
Claims 4-6 are rejected under 35 U.S.C. 103 as being unpatentable over Watts et al. US Publication 2013/0169844 and Lin et al. US Publication 2015/0294490 as applied to claim 3 above, and further in view of Wong et al. US Publication 2021/0004933 (hereafter “Wong”).
Referring to claim 4, Watts discloses wherein the at least one processor is further configured to execute the instructions to:
determine a parameter of the geometric transformation that decreases an error between a position of the feature point of the RROI of the first image and a transformed position of the feature point of the RROI of the second image according to the geometric transformation image (paragraph 52, Operation 604 involves determining an object center line of the object in each initial image as described above with reference to FIGS. 4A and 4B. In some embodiments, other alignment and/or scaling techniques may be used during operation 604).
Watts does not disclose expressly using a gradient descent.
Wong discloses wherein the at least one processor is further configured to execute the instructions to:
determine a parameter of the geometric transformation using a gradient descent that decreases an error between a position of the feature point of the RROI of the first image and a transformed position of the feature point of the RROI of the second image according to the geometric transformation (paragraph 168, An optimizer can be used (e.g. an energy-based gradient descent optimizer, a linear programming optimizer, etc.) to adjust the vertices of the mesh, optimizing for feature alignment between images and other constraints).
At the time of the effective filing date of the claimed invention, it would have obvious to a person of ordinary skill in the art to use a parameter of the geometric transformation using a gradient descent. The motivation for doing so would have been to increase the accuracy of aligning images. Therefore, it would have been obvious to combine Wong with Watts to obtain the invention as specified in claim 4.
Referring to claim 5, Watts discloses wherein the parameter of the geometric transformation includes at least one of a translation parameter, a scaling parameter, or a perspective change parameter (paragraph 52, Operation 604 involves determining an object center line of the object in each initial image as described above with reference to FIGS. 4A and 4B. In some embodiments, other alignment and/or scaling techniques may be used during operation 604).
Referring to claim 6, Watts discloses wherein the perspective change parameter is defined based on a relative position of the first image sensor and the second image sensor (paragraph 50, The stereo disparity may be used during detecting operation 602 to determine proximity of each pixel or patch in the stereo images to the camera and therefore to identify at least the background portion of the image).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PETER K HUNTSINGER whose telephone number is (571)272-7435. The examiner can normally be reached Monday - Friday 8:30 - 5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Benny Q Tieu can be reached at 571-272-7490. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PETER K HUNTSINGER/Primary Examiner, Art Unit 2682