Detailed Action
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments with respect to claims 1-26 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
PNG
media_image1.png
308
650
media_image1.png
Greyscale
On page 11, Applicant argues,
Note that the new ground of rejection for claim 1 relies on the combination of Lawrence in view of Pertsel. Pertsel, columns 6 and 7, lines 20-67 and 1-15, respectively, teaches using motion values for objects to control exposure parameters. Thus, when combined with the global compensated object tracking taught by Lawrence, Pertsel provides teachings which covers the claim limitation “selecting a value for at least on be image capture parameter associated with the plurality of image frames based on the motion vectors correspond to the difference in the adjusted position of the at least one object to control capture of at least on subsequent image frame”.
Specifically, Lawrence provides the “motion vectors correspond to the difference in the adjusted position of the at least one object” and Pertsel provides “selecting a value for at least on be image capture parameter… to control capture of at least on subsequent image frame”. Therefore, the combination of Lawrence in view of Pertsel teaches all the limitations recited in claim 1 (see below for additional details).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3, 6, 13, 14, 16, 19, and 26 are rejected under 35 U.S.C. 103 as being unpatentable over Lawrence et al. (US 20170134746 A1), (hereinafter, Lawrence) in view of Pertsel et al. (US 8189057 B2), (hereinafter, Pertsel).
Regarding claim 1, Lawrence teaches a method for processing image data, the method comprising:
determining, based on data from one or more sensors associated with an image capture device, a movement of the image capture device associated with a capture of a plurality of image frames (Lawrence, “Global motion estimation may be performed in different ways, depending on the implementation. In one example, global motion parameters are estimated from a sampled motion vector field. Gradient descent or another regression method may be used to select appropriate motion vectors that correspond to camera shake and remove motion vector outliers.”, pg. 2, paragraph 0020, “Depending on its applications, computing device 100 may include other components that may or may not be physically and electrically coupled to the board 2. These other components include, but are not limited to… an accelerometer (not shown), a gyroscope (not shown), a speaker 30, a camera 32, a microphone array 34, and a mass storage device (Such as hard disk drive) 10, compact disk (CD) (not shown), digital versatile disk (DVD) (not shown), and so forth).”, pg. 4, paragraph 0042, Global motion, corresponding to the shake of the camera, is determined for captured frames by sampling and analyzing the motion vector field of a video stream through a camera sensor.);
performing compensation of the movement of the image capture device to generate a plurality of compensated image frames by adjusting a position of at least one object in each of the plurality of image frames based on the movement of the image capture device determined based on the data from the one or more sensors; and determining a motion of the at least one object based on motion vectors corresponding to a difference in the adjusted position of the at least one object among the plurality of compensated image frames (Lawrence, “FIGS. 5 and 6 show the third approach mentioned above as process flow diagrams. In this example, object motion vectors are isolated and then used to fine-tune the global motion vectors. In this case, object motion vectors 410 are used to realign 430 the ROI after global motion has been compensated 418… In general global motion vectors are first used to estimate global motion 404 due to camera shake and used to determine a gross ROI.”, pg. 3, paragraphs 0034-0035, “The output of the ROI coordinates adjustment block is fed to a decoder 422 which then provides the stabilized video 424. The ROI adjustment block first adjusts the ROI coordinates to minimize camera shake 418 as in FIG. 2. This uses the camera shake compensation 414 determined from the global motion compensation 404. Then using the stabilized coordinates 418, the ROI coordinates are adjusted 430 to minimize the deviation of the object of interest, based on the object tracking 408, from the center of the ROI.”, pg. 3, paragraph 0037, see Figs. 5 and 6, Global motion compensation is then performed for the frames, which adjusts a position or ROI coordinates corresponding to the frames including objects for tracking. After this global compensation, object motion vectors are determined for the adjusted frames and then used to realign the ROI for video stabilization.).
Lawrence does not teach selecting a value for at least one image capture parameter associated with the plurality of image frames based on the motion vectors corresponding to the difference in the adjusted position of the at least one object to control capture of at least one subsequent image frame.
However, Pertsel teaches selecting a value for at least one image capture parameter associated with the plurality of image frames based on the motion vectors corresponding to the difference in the adjusted position of the at least one object to control capture of at least one subsequent image frame (Pertsel, “In the step 73, data of N number of pre-capture images are used to calculate motion quantities for use in setting the exposure parameters, where N equals two or more, and can be five or more. As explained in detail below, any change in motion of the scene image relative to the camera's photosensor is detected and quantified by looking at changes in Successive pre-capture images, both globally (movement of the entire image) and locally (local movement within the image)... A next step 81 determines whether the exposure parameters automatically calculated in the step 75 are such that the motion quantities will not cause them to be altered. For example, if the exposure duration (shutter speed) is set by the step 75 to be below a certain threshold, then no further decrease of the exposure time to reduce motion blur should be done.”, column 6 and 7, lines 40-67 and 1-22, respectively, Global and local motion vectors are determined for a set of image frames and used to adjust exposure settings to reduce motion blur.).
Lawrence teaches performing global motion compensation for a set of video frames to enable accurate detection and tracking of local object motion vector for video stabilization (Lawrence, paragraphs 0036-0037, see Figs. 5 and 6). Lawrence does not teach using these object motion vectors to set image capture parameters for subsequent image frames. Pertsel teaches adjusting image capture parameters, such as exposure time, for subsequent frames based on detected object motion (see above). Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified the local object motion vector detection of Lawrence to be used for adjusting image capture parameters as taught by Pertsel (Pertsel, column 6 and 7, lines 40-67 and 1-2). The motivation for doing so would have been to adjust the exposure settings with respect to local object motion, thereby reducing motion blur for specific objects of interest in the image (as suggested by Pertsel, “It is often difficult for the user to hold a camera by hand during an exposure without imparting some degree of shake or jitter, particularly when the camera is very Small and light. As a result, the captured image may have a degree of overall motion blur that depends on the exposure time, the longer the time the more motion blur in the image. In addition, long exposures of a scene that is totally or partially moving can also result in motion blur in the captured image. An object moving fast across the scene, for example, may appear blurred in the image.”, column 2, lines 28-37). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine the teachings of Lawrence with Pertsel to obtain the invention as specified in claim 1.
Regarding claim 2, Lawrence in view of Pertsel teaches the method of claim 1, wherein the one or more sensors include at least one of a gyroscope, an accelerometer, a magnetometer, and an inertial measurement unit (IMU) (Lawrence, “Depending on its applications, computing device 100 may include other components that may or may not be physically and electrically coupled to the board 2. These other components include, but are not limited to… an accelerometer (not shown), a gyroscope (not shown), a speaker 30, a camera 32, a microphone array 34, and a mass storage device (Such as hard disk drive) 10, compact disk (CD) (not shown), digital versatile disk (DVD) (not shown), and so forth).”, pg. 4, paragraph 0042).
Regarding claim 3, Lawrence in view of Pertsel teaches the method of claim 1. Lawrence in view of Pertsel, as presented above, does not teach wherein the movement of the image capturing device occurs during an exposure time corresponding to each of the plurality of image frames.
However, Pertsel further teaches wherein the movement of the image capturing device occurs during an exposure time corresponding to each of the plurality of image frames (Perstel, “It is often difficult for the user to hold a camera by hand during an exposure without imparting some degree of shake or jitter, particularly when the camera is very Small and light. As a result, the captured image may have a degree of overall motion blur that depends on the exposure time, the longer the time the more motion blur in the image.”, column 2, lines 28-33, “Rather than post-processing the acquired video data by taking image motion into account, however, the present invention monitors images of the scene in advance of taking the picture and then sets the exposure parameters to values that enhance the resulting image based on the amount of motion present. The processing calculates at least an optimal exposure time that can be used along with other exposure parameters to acquire data of an image... In FIG. 1, an example of a camera in which the present invention may be implemented is schematically shown, which may be a still camera or a video camera.”, column 4, lines 29-49, Images are taken using an optical camera to capture exposures during camera movement. The camera’s motion is detected and used to control exposure time for each frame.).
Lawrence in view of Pertsel teaches capturing sequential image frames of a compressed video stream and performing global motion compensation for movement of the camera sensor (Lawrence, paragraphs 0026 and 0035). Lawrence in view of Pertsel does not teach that this movement occurs during an exposure time for each of the captured frames. Pertsel further teaches detecting movement of an optical camera during an exposure time (see above). Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified the camera sensor of Lawrence in view of Pertsel to capture and compensate global motion during an exposure time as further taught by Pertsel (Pertsel, column 4, lines 29-49, see Fig. 1). The motivation for doing so would have been to account for camera shake specifically during the exposure period for each frame, thereby increasing the accuracy of motion vector extraction and stabilization. Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine the teachings of Lawrence in view of Pertsel with the further teachings of Pertsel above to obtain the invention as specified in claim 3.
Regarding claim 13, Lawrence in view of Pertsel teaches the method of claim 1, wherein the at least one image capture parameter includes at least one of an exposure time and a gain (Pertsel, “However, in most situations the scene is not so brightly illuminated. Therefore, when the preliminary parameters calculated by the step 75 are not within optimum ranges, they are adjusted by a step 85 in order to optimize them for the amount of motion that was calculated by the step 73. Generally, if that
motion is high, the exposure time is reduced, with a corresponding increase in the size of the aperture and/or increase in the gain in order to maintain the same average image signal luminescence.”, column 7, lines 34-42).
Claim 14 corresponds to claim 1, with the addition of an apparatus comprising a memory and a processor configured to execute the method according to claim 1. Lawrence in view of Pertsel teaches the addition of an apparatus comprising a memory and a processor (Lawrence, “FIG. 7 is a block diagram of a single computing device 100 in accordance with one implementation. The computing device 100 houses a system board 2. The board 2 may include a number of components, including but not limited to a processor 4 and at least one communication package 6.”, pg. 4, paragraph 0041, lines 1-6) configured to execute the method according to claim 1. As indicated in the analysis of claim 1, Lawrence in view of Pertsel teaches all the limitations according to claim 1. Therefore, claim 14 is rejected for the same reasons of obviousness as claim 1.
Claim 15, 16, and 26 corresponds to claims 2, 3, and 13, respectively, with the addition of an apparatus comprising a memory and a processor configured to execute the method according to claims 2, 3, and 13, respectively. Lawrence in view of Pertsel and further in view of Buyukozturk teaches the addition of an apparatus comprising a memory and a processor (see analysis of claim 14) configured to execute the method according to claims 2, 3, and 13, respectively. As indicated in the analysis of claims 2, 3, and 13, Lawrence in view of Pertsel and further in view of Buyukozturk teaches all the limitations according to claims 2, 3, and 13, respectively. Therefore, claims 15, 16, and 26 are rejected for the same reasons of obviousness as claims 2, 3, and 13, respectively.
Claims 4 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Lawrence et al. (US 20170134746 A1) in view of Pertsel et al. (US 8189057 B2) and further in view of Gao et al. (US 20210112188 A1), (hereinafter, Gao).
Regarding claim 4, Lawrence in view of Pertsel teach the method of claim 1. Lawrence in view of Pertsel does not teach further comprising: in response to determining that the movement of the image capturing device is greater than a threshold value, selecting a default value for the at least one image capture parameter.
However, Gao teaches in response to determining that the movement of the image capturing device is greater than a threshold value, selecting a default value for the at least one image capture parameter (Gao, “In general, multiple features may be combined to evaluate the extent to which environmental brightness measurements are caused by reflected light from the display, which may indicate a need to reduce the rate of exposure change… In further examples, sensor data from a gyroscope may be used to evaluate device motion in addition to or instead of camera image data. It may be advantageous to use both a gyroscope and image-based motion detection because
a gyroscope provides a measurement of global motion while image-based motion detection provides a measurement of local motion. If sensor data from a gyroscope is used, then a control system may refrain from decreasing the rate of exposure change when the gyroscope indicates a significant
amount of device motion (e.g., above a high threshold level).”, pg. 5, paragraphs 0053 and 0054, Global motion caused by movement of a camera is determined using a gyroscope. Thresholding is applied to decide if exposure changes should be reduced or left at the normal rate. When motion is determined to be above this threshold, the system keeps the default exposure-change behavior.).
Lawrence in view of Pertsel teaches estimating global and local motion to control exposure settings to reduce motion blur (Pertsel, column 6 and 7, lines 40-67 and 1-22, respectively). Gao teaches applying thresholding to global motion to control exposure settings (see above). Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified the exposure control of Lawrence in view of Pertsel to additionally control exposure setting based on thresholding global motion as taught by Gao (Gao, pg. 5, paragraphs 0053 and 0054). The motivation for doing so would have been to maintain stable exposure control during large camera movements. Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine the teachings of Lawrence in view of Pertsel with Gao to obtain the invention as specified in claim 4.
Claim 17 corresponds to claim 4, with the addition of an apparatus comprising a memory and a processor configured to execute the method according to claim 4. Lawrence in view of Pertsel and further in view of Gao teaches the addition of an apparatus comprising a memory and a processor (see analysis of claim 14) configured to execute the method according to claim 4. As indicated in the analysis of claim 4, Lawrence in view of Pertsel and further in view of Gao teaches all the limitations according to claim 4. Therefore, claim 17 is rejected for the same reasons of obviousness as claim 4.
Claims 5 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Lawrence et al. (US 20170134746 A1) in view of Pertsel et al. (US 8189057 B2) and further in view of Smith et al. (“Electronic image stabilization using optical flow with inertial fusion”, IEEE/RSJ International Conference on Intelligent Robots and Systems, 2010), (hereinafter, Smith).
Regarding claim 5, Lawrence in view of Pertsel teach the method of claim 1. Lawrence in view of Pertsel does not teach wherein adjusting the position of the at least one object includes computing an electronic image stabilization (EIS) compensation.
However, Smith teaches wherein adjusting the position of the at least one object includes computing an electronic image stabilization (EIS) compensation (Smith, “This paper presents a novel EIS algorithm designed to operate in the presence of large image displacement, image blurring, and moving objects. Using the similarity motion model, the algorithm fuses pyramidal Lucas-Kanade optical flow using Shi-Tomasi good features with inertial measurement motion estimation by way of a discrete Kalman filter. Inertial measurement motion estimation is performed by summing angular displacements between frames of a MIDG II inertial measurement unit (IMU) and multiplying the angular displacements by a constant. The two motion estimates are then optimally fused using a nine-state discrete Kalman filter.”, pg. 1, 2nd column, 1st full paragraph, see Section III. Optical Flow with Inertial Fusion and Fig. 6).
Lawrence in view of Pertsel teaches performing global motion tracking and compensation using any suitable technique to enhance the frame for motion tracking (Lawrence, paragraphs 0024 and 26). Smith teaches performing global motion compensation using an electronic image stabilization algorithm which combines optical flow with inertial measurements (see above). Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified Lawrence in view of Pertsel by replacing the global motion compensation with the electronic image stabilization algorithm as taught by Smith (Smith, pg. 1, 2nd column, 1st full paragraph, see Section III. Optical Flow with Inertial Fusion and Fig. 6). The motivation for doing so would have been to reduce prediction error compared to optical flow alone (as suggested by Smith, “The novel algorithm presented in this paper, optical flow with inertial fusion, combines these two methods, and is capable of reduction in RMS error compared to 40% optical flow alone in the presence of moving objects.”, pg. 8, 2nd column, 3rd full paragraph, lines 6-9). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine the teachings of Lawrence in view of Pertsel with Smith to obtain the invention as specified in claim 5.
Claim 18 corresponds to claim 5, with the addition of an apparatus comprising a memory and a processor configured to execute the method according to claim 5. Lawrence in view of Pertsel and further in view of Smith teaches the addition of an apparatus comprising a memory and a processor (see analysis of claim 14) configured to execute the method according to claim 5. As indicated in the analysis of claim 5, Lawrence in view of Pertsel and further in view of Smith teaches all the limitations according to claim 5. Therefore, claim 18 is rejected for the same reasons of obviousness as claim 5.
Claims 6 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Lawrence et al. (US 20170134746 A1) in view of Pertsel et al. (US 8189057 B2) and further in view of Kale et al. (“Moving Object Tracking using Optical Flow and Motion Vector Estimation”, 4th internation conference on reliability Infocom technologies and optimization, IEEE, 2015), (hereinafter, Kale).
Regarding claim 6, Lawrence in view of Pertsel teaches the method of claim 1. Lawrence in view of Pertsel does not teach wherein determining the motion of the at least one object includes determining an optical flow between the plurality of image frames.
However, Kale teaches wherein determining the motion of the at least one object includes determining an optical flow between the plurality of image frames (Kale, “In the proposed system, optical flow is used in object detection stage… Motion vectors estimation technique is used in object tracking stage. It eliminates the shortcomings in tracking due to conventional optical flow as mentioned in [6].”, pg. 1, 2nd column, 1st full paragraph, “The principal objective of optical flow estimation is to separate the moving foreground objects from the background and generate optical flow field vector for the moving object. Optical flow calculates the motion between two frames which are taken at different time intervals for every pixel in the frame.”, pg. 2, section II. Proposed System, A. Object detection using Optical flow, lines 1-6, “There are various methods available in optical flow estimation. Among these, phase based methods provides good results but they are complex to implement. Hence differential methods like Horn and Schunk, Lucas and Kanade are used.”, pg. 3, 1st column, 1st full paragraph, lines 1-4, see Fig. 2 and 3).
Lawrence in view of Pertsel teaches identifying object motion vectors using any suitable object identification and motion tracking technique (Lawrence, paragraph 0025). Kale teaches performing optical flow motion tracking for objects (see above). Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified the object motion tracking of Lawrence in view of Pertsel to use the suitable optical flow techniques taught by Kale (Kale, pg. 2, section II. Proposed System, A. Object detection using Optical flow, lines 1-6, pg. 3, 1st column, 1st full paragraph, lines 1-4, see Fig. 2 and 3). The motivation for doing so would have been to increase accuracy of object tracking in cases of blurry or cluttered backgrounds (as suggested by Kale, “The optical flow gives valuable information about the object movement even if no quantitative parameters are computed. The motion vector estimation technique can provide an estimation of object position from consecutive frames which increases the accuracy of this algorithm and helps to provide robust result irrespective of image blur and cluttered background.”, see abstract). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine the teachings of Lawrence in view of Pertsel with Kale to obtain the invention as specified in claim 6.
Claim 19 corresponds to claim 6, with the addition of an apparatus comprising a memory and a processor configured to execute the method according to claim 6. Lawrence in view of Pertsel and further in view of Kale teaches the addition of an apparatus comprising a memory and a processor (Lawrence, “FIG. 7 is a block diagram of a single computing device 100 in accordance with one implementation. The computing device 100 houses a system board 2. The board 2 may include a number of components, including but not limited to a processor 4 and at least one communication package 6.”, pg. 4, paragraph 0041, lines 1-6) configured to execute the method according to claim 6. As indicated in the analysis of claim 6, Lawrence in view of Pertsel and further in view of Kale teaches all the limitations according to claim 6. Therefore, claim 19 is rejected for the same reasons of obviousness as claim 6.
Claims 7-11 and 20-24 are rejected under 35 U.S.C. 103 as being unpatentable over Lawrence et al. (US 20170134746 A1) in view of Pertsel et al. (US 8189057 B2) and further in view of Ichihashi et al. (US 20100119176 A1), (hereinafter, Ichihashi).
Regarding claim 7, Lawrence in view of Pertsel teaches the method of claim 1. Lawrence in view of Pertsel does not teach wherein determining the motion of the at least one object includes determining a motion mask between a first image frame and a second image frame from the plurality of image frames.
However, Ichihashi teaches wherein determining the motion of the at least one object includes determining a motion mask between a first image frame and a second image frame from the plurality of image frames (Ichihashi, “ In step S43, the mask generating unit 33 generates a motion mask using the input image supplied from the upsampling unit 31 and the estimation image supplied from the motion compensation unit 32. Thereafter, the mask generating unit 33 supplies the generated motion mask to the mixing unit 34 and the weight computing unit 43… In step S44, the mixing unit 34 mixes the input
image supplied from the upsampling unit 31 with the estimation image supplied from the motion compensation unit 32 using the motion mask supplied from the mask generating unit 33. That is, the mixing unit 34 selects one of the pixels of an image to be obtained from now (hereinafter also referred to as an ’SR mixing image’) as a pixel of interest. The mixing unit 34 then computes, using the pixel value of a pixel of the motion mask located at a position the same as that of the pixel of interest, a weight Wi (0s Wis1) of a pixel in the input image located at a position the same as that of the pixel of interest and a weight We (=1-Wi) of a pixel in the estimation image located at a position the same as that of the pixel of interest.”, pgs. 5 and 6, paragraphs 0095-0098, A motion mask is generated from motion estimation between images. This mask is used to generate blending weights for a super-resolution process.).
Lawrence in view of Pertsel teaches tracking local motion of an object using globally compensated images (Lawrence, paragraphs 0035-0037). Ichihashi teaches generating a motion mask which is used to compute blending weights for image super-resolution (see above). Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified the local motion tracking of Lawrence in view of Pertsel to include a motion mask for image super-resolution as taught by Ichihashi (Ichihashi, pgs. 5 and 6, paragraphs 0095-0098). The motivation for doing so would have been to increase the resolution of the image based on motion of the object, thereby increasing the quality of the image (as suggested by Ichihashi, “That is, when a higher-resolution output image is acquired from the input image, the image quality of the entire area of the output image can be easily and reliably increased.”, pg. 10, paragraph 0177, lines 6-8). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine the teachings of Lawrence in view of Pertsel with Ichihashi to obtain the invention as specified in claim 7.
Regarding claim 8, Lawrence in view of Pertsel and further in view of Ichihashi teaches the method of claim 7, wherein the motion mask includes one or more motion vectors indicating a shift of at least one pixel between the first image frame and the second image frame (Ichihashi, “That is, the motion compensation unit 32 detects a motion vector of each of the pixels of the SR image using the input image and the SR image. Thereafter, the motion compensation unit 32 performs motion compensation on the SR image using the detected motion vectors and generates an estimation image.… In step S43, the mask generating unit 33 generates a motion mask using the input image supplied from the upsampling unit 31 and the estimation image Supplied from the
motion compensation unit 32.”, pg. 5, paragraphs 0094 and 0095, A motion vector is determined and used to generate an estimation image. This estimation image is compared with the input image to generate the motion mask.).
Regarding claim 9, Lawrence in view of Pertsel and further in view of Ichihashi teaches the method of claim 8, further comprising: determining a weighted table that includes one or more weight values corresponding to at least a portion of the one or more motion vectors (Ichihashi, “the mixing unit 34 selects one of the pixels of an image to be obtained from now (hereinafter also referred to as an “SR mixing image') as a pixel of interest. The mixing unit 34 then computes, using the pixel value of a pixel of the motion mask located at a position the same as that of the pixel of interest, a weight Wi (0s Wis1) of a pixel in the input image located at a position the same as that of the pixel of interest and a weight We (=1-Wi) of a pixel in the estimation image located at a position the same as that of the pixel of interest.”, pg. 6, paragraph 0098, The motion vectors are used to determine the motion mask. From this, weights can be determined on a per-pixel basis in order to blend in the super-resolution process.).
Regarding claim 10, Lawrence in view of Pertsel and further in view of Ichihashi teaches the method of claim 9, wherein the one or more weight values are selected based on a central region in the plurality of image frames (Ichihashi, “the mixing unit 34 selects one of the pixels of an image to be obtained from now (hereinafter also referred to as an “SR mixing image') as a pixel of interest. The mixing unit 34 then computes, using the pixel value of a pixel of the motion mask located at a position the same as that of the pixel of interest, a weight Wi (0s Wis1) of a pixel in the input image located at a position the same as that of the pixel of interest and a weight We (=1-Wi) of a pixel in the estimation image located at a position the same as that of the pixel of interest.”, pg. 6, paragraph 0098, Weights are selected based on matching pixel points across the images. This includes selecting weights for central pixels of the image.).
Regarding claim 11, Lawrence in view of Pertsel and further in view of Ichihashi teaches the method of claim 9, wherein the one or more weight values are selected based on a region of interest in the plurality of image frames (Ichihashi, “Accordingly, for example, as shown in FIG. 12, the weight computing unit 43 increases the weight Wh of an enhancement image for a pixel of an area R11 of a moving subject for which the advantage of Super-resolution processing is not obtained and a pixel of an area R12 including a diagonal line for which the edge enhancement processing is
effective… For example, the pixel value of a pixel of a motion mask indicates the level of motion of the subject. Accordingly, it is determined that a pixel having a pixel value larger than or equal to a predetermined threshold value is a pixel of the area of the moving subject. Thus, the weight Wh of a pixel of the enhancement image located at a position the same as that of the pixel is further increased.”, pg. 10, paragraphs 0172-0174, see Fig. 12, Weights can be selected and adjusted according to objects of interest.).
Claims 20, 21, 22, 23, and 24 corresponds to claims 7, 8, 9, 10 and 11, respectively, with the addition of an apparatus comprising a memory and a processor configured to execute the method according to claims 7, 8, 9, 10 and 11, respectively. Lawrence in view of Pertsel and further in view of Ichihashi teaches the addition of an apparatus comprising a memory and a processor (see analysis of claim 14) configured to execute the method according to claims 7, 8, 9, 10 and 11, respectively. As indicated in the analysis of claim 7, 8, 9, 10 and 11, Lawrence in view of Pertsel and further in view of Ichihashi teaches all the limitations according to claims 7, 8, 9, 10 and 11, respectively. Therefore, claim 20, 21, 22, 23, and 24 is rejected for the same reasons of obviousness as claims 7, 8, 9, 10 and 11, respectively.
Claims 12 and 25 are rejected under 35 U.S.C. 103 as being unpatentable over Lawrence et al. (US 20170134746 A1) in view of Pertsel et al. (US 8189057 B2) and further in view of Ichihashi et al. (US 20100119176 A1) and Hamamoto et al. (JP 2011155582 A), (hereinafter, Hamamoto).
Regarding claim 12, Lawrence in view of Pertsel and further in view of Ichihashi teaches the method of claim 11. Lawrence in view of Pertsel and further in view of Ichihashi does not teach wherein a portion of the one or more weight values corresponding to an area outside the region of interest is set to zero.
However, Hamamoto teaches wherein a portion of the one or more weight values corresponding to an area outside the region of interest is set to zero (Hamamoto, “as shown in FIG. 7A, when the size of the main subject area T11 is equal to or larger than the threshold value, the weight of the motion vector detected in the small area including 20 the background area may be set to zero. Similarly, as shown in FIG. 7B, when the size of the main subject region T12 is smaller than the threshold, the weight of the motion vector detected in the small region including the main subject region T12 may be set to zero.”, pg. 16, lines 19-23, Weights for background regions are set to zero when the size of a subject meets or exceeds a threshold value.).
Lawrence in view of Pertsel and further in view of Ichihashi teaches selecting and adjusting weights for objects in images (Ichihashi, “Accordingly, for example, as shown in FIG. 12, the weight computing unit 43 increases the weight Wh of an enhancement image for a pixel of an area R11 of a moving subject… ”, pg. 10, paragraphs 0172-0174, see Fig. 12). Hamamoto teaches setting background region weights to zero based on a subject size thresholding (see above). Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified the weights selection and adjustment of Lawrence in view of Pertsel and further in view of Ichihashi to include the subject size thresholding as taught by Hamamoto (Hamamoto, pg. 16, lines 19-23), thereby setting background region weights to zero. The motivation for doing so would have been to filter out redundant weights corresponding to background regions, thereby increasing processing speed. Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine the teachings of Lawrence in view of Pertsel and further in view of Ichihashi with Hamamoto to obtain the invention as specified in claim 12.
Claim 25 corresponds to claim 12, with the addition of an apparatus comprising a memory and a processor configured to execute the method according to claim 12. Lawrence in view of Pertsel and further in view of Ichihashi and Hamamoto teaches the addition of an apparatus comprising a memory and a processor (see analysis of claim 14) configured to execute the method according to claim 12. As indicated in the analysis of claim 12, Lawrence in view of Pertsel and further in view of Ichihashi and Hamamoto teaches all the limitations according to claim 12. Therefore, claim 25 is rejected for the same reasons of obviousness as claim 12.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CONNOR LEVI HANSEN whose telephone number is (703)756-5533. The examiner can normally be reached Monday-Friday 9:00-5:00 (ET).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sumati Lefkowitz can be reached at (571) 272-3638. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CONNOR L HANSEN/Examiner, Art Unit 2672
/SUMATI LEFKOWITZ/Supervisory Patent Examiner, Art Unit 2672
Prosecution Insights