DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments, see Remarks at page 8, filed 30 December 2025, with respect to the objections to claims 4 and 14 for minor informalities have been fully considered and are persuasive. The objections have been withdrawn.
Applicant’s arguments, see Remarks at page 8, filed 30 December 2025, with respect to the rejections of claims 5, 9, 15 and 19 under 35 U.S.C. 112(b) have been fully considered and are persuasive. The rejections have been withdrawn.
Applicant’s arguments, see Remarks at pages 8-11, filed 30 December 2025, with respect to the rejections of claims 1-4, 8-14 and 18-20 under 35 U.S.C. 103 have been fully considered but are not persuasive.
Applicant argues “there is no reasonable combination of the applied references that would meet all of the claim’s features”, and in particular, “arranging the plurality of generated residual images into a micro image based on the depth map”. Applicant states, “Monteiro discloses a process for updating a micro-image buffer from decoded sub-aperture images (SAIs) and separately calculating a prediction residue, not for “arranging the plurality of generated residual images into a micro image based on the depth map.” FIG. 2 of Monteiro demonstrates that the residue is an output, not an input used to generate the microimage buffer ... As shown in FIG. 2 of Monteiro, the microimage data is stored in the ‘MI Buffer’ (which is populated by ‘Decoded SAIs’), while the residue is calculated separately as the ‘CB Residue’ after the prediction step” (original emphasis).
Examiner respectfully disagrees with Applicant’s argument that there is no obvious combination of Chen and Monteiro to teach “arranging the plurality of generated residual images into a micro image based on the depth map”.
Chen discloses encoding LF data into a compressed version of the full light field in SAI representation. As shown in the decoder section of the block diagram in Figure 2, the fully-decoded compressed output is produced from two inputs: the re-ordered SAI residual sequence and the predicted LF from the disparity guided sparse coding. The decoded residuals are put back into the proper order by the decoder, i.e., “arranged”, which outputs the compressed version of the input SAI-based LF data. The residuals are generated, in part, by the disparity guided sparse coding in the encoder, and are thus “based on the depth map.” Therefore, Chen discloses arranging the plurality of generated residual images into a compressed full light field (a compressed version of the SAI-based input LF data), based on the depth map. However, Chen does not explicitly disclose arranging the residuals into a micro image.
Monteiro discloses arranging light field data into a micro image representation using hybrid light field coding of LF data with “MI- or SAI-based data representations” (See section III.A). The LF prediction module, shown in Figure 2, switches between different prediction modes, including intra-SAI, inter-SAI, intra-MI, and inter-MI. This is noted to demonstrate that SAI and MI are two types of LF data representations that are not only used together in the prior art, but have a direct correspondence between each other. The correspondence is a bidirectional pixel correspondence or conversion between SAI and MI representations, as shown in Figures 3 and 4. This indicates that both types of representations would be considered by a POSITA when designing an encoder for LF data and also that organizing sub-aperture views into a micro image representation was known at the time.
Applicant argues that the residual images generated by Monteiro’s LF prediction module are generated after the MI representation conversion takes place, and thus cannot describe an arrangement of residuals images into a micro image. While Applicant’s summary of various features of Monteiro may indeed be accurate, under the broadest reasonable interpretation of the claims the combination of Chen in view of Monteiro teaches “arranging the plurality of generated residual images into a micro image based on the depth map”.
The combination of Chen in view of Monteiro applied to claims 1 and 11 appends the SAI-to-MI conversion of Monteiro to the output of Chen’s decoder (a compressed version of the input SAI-based LF data), thereby producing the light field as MI-based LF image data. Thus, a micro image is generated from the arranged plurality of residual images based on the depth map.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed inventions absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-4, 8-14 and 18-20 are rejected under 35 U.S.C. 103 as being unpatentable over Light Field Compression with Disparity Guided Sparse Coding based on Structural Key Views to Chen in view of Light Field Image Coding Based on Hybrid Data Representation to Monteiro.
Regarding claim 1, Chen teaches an image processing method comprising:
obtaining a plurality of main images (Chen, section III.A, “a group of Structural Key Views (SKV) that best represent the spatial information of the LF”; FIG. 2(a), “Decoded SKV”) for a plurality of light field images captured from different viewpoints (Chen, section I.A, “Each sub-view index pair (s,t) corresponds to a unique viewing angle.”; The “plurality of light field images” corresponds to all of the original views/viewpoints.);
generating a depth map (a disparity map represents depth as an amount of shift between two images) representing depth information of the plurality of light field images (Chen, section I.A, “Based on the above characteristics, a pixel-wise disparity map (motion vectors) can be efficiently calculated for the LF, and just one disparity map will be able to describe the parallax of all pixels across all SVIs.”), based on the plurality of main images (See FIG. 2 of Chen. The disparity map is generated representing depth information of the input views, which includes the sampled/main views. Disparity corresponds with depth.);
generating a plurality of prediction images for the plurality of light field images (Chen, FIG. 2, “Predicted LF”), based on the plurality of main images (The predicted LF is generated from the sampled views.);
generating a plurality of residual images representing a difference between the plurality of light field images and the plurality of prediction images (See FIG. 2 of Chen. The residual images are obtained as the difference between the input LF data and the predicted LF images.); and
arranging the plurality of generated residual images into an image based on the depth map (The plurality of generated residual images are re-ordered with the predicted LF data, i.e., arranged, into a full light field, i.e., all views in SAI-representation, based on the depth map. See FIG. 2 of Chen.), but does not teach that which is explicitly taught by Monteiro.
Monteiro teaches generating a micro image (Monteiro, pg. 115731, section III.A, “The main feature of the 4D LF data representation is that it allows the conversion between MI- and SAI-based data representations to become seamless and reversible”; FIGs. 3 and 4 of Monteiro show the relationship and conversion between each SAI and the MI representation.).
Chen discloses a light field (LF) compression method using a sub-aperture image-based representation (SAI) of LF data. Thus, Chen shows that it was known in the art before the effective filing date of the claimed invention to use SAI-based encoding to compress light field image data, which is analogous to the claimed invention in that it is pertinent to the problem being solved by the claimed invention, improving light field image compression efficiency. Monteiro discloses a hybrid light field coding method using a sub-aperture image-based representation (SAI) and a micro-image based representation (MI) of LF data, and also discloses spiral scanning to convert a set of decoded sub-views into a micro image (FIG. 3). Thus, Monteiro shows that it was known in the art before the effective filing date of the claimed invention to use SAI and MI-based encoding together in a single process of compressing LF data, which is analogous to the claimed invention in that it is pertinent to the problem being solved by the claimed invention, improving light field image compression efficiency.
A person of ordinary skill in the art would have been motivated to convert the LF data output by the decoder disclosed by Chen to a micro image (MI) representation as disclosed by Monteiro, to thereby use the SC-SKV codec and then apply the MI conversion to ultimately output compressed LF data in MI representation. Based on the foregoing, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have made such modification according to known methods to yield the predictable results to have the benefit of producing compressed LF data that is in a native format of plenoptic cameras, thereby enabling direct and quicker access to the data on such cameras for playback or editing.
Regarding claim 2, Chen in view of Monteiro teaches the image processing method of claim 1, wherein the generating of the depth map comprises:
identifying a second pixel of a second main image corresponding to a first pixel of a first main image (Chen, section I.A., “a pixel-wise disparity map (motion vectors) can be efficiently calculated for the LF, and just one disparity map will be able to describe the parallax of all pixels across all SVIs.”; Disparity is the displacement between corresponding pixels in the SVIs.); and
generating the depth map based on a difference (the amount of displacement, represented by the motion vector) between a location of the first pixel and a location of the second pixel, wherein each of the first main image and the second main image is one of the plurality of main images (Chen, section I.A, “just one disparity map will be able to describe the parallax of all pixels across all SVIs.”).
Regarding claim 3, Chen in view of Monteiro teaches the image processing method of claim 1, further comprising adjusting a value of the depth map to an integer (Chen, section IV.E, “linear operator that quantizes disparity values into integers that correspond to the disparity segment indices in D”; The LF dictionary is generated from the disparity map. To reconstruct the LF image, the disparity values are adjusted to integers.).
Regarding claim 4, Chen in view of Monteiro teaches the image processing method of claim 1, wherein the arranging the plurality of generated residual images (which is a portion of the overall sequence that generates the micro image) comprises:
generating a synthesized image (See Chen at FIG. 2; Disparity is used to approximate synthetic SAIs for the non-sampled SKVs.) by arranging the plurality of residual images to correspond to different viewpoints at which the plurality of light field images are captured (Chen, section III.A, “the SVI residual sequence from the approximation in the previous step is reordered”; Each SVI corresponds to a different viewpoint.); and
generating the micro image (Monteiro, pg. 115731, section III.A, “The main feature of the 4D LF data representation is that it allows the conversion between MI- and SAI-based data representations to become seamless and reversible.”) by arranging the synthesized image by using the depth map (The converted MI results from the prior processing, i.e., the arranging, including the use of the depth map.).
The rationale for obviousness is the same as provided for claim 1.
Regarding claim 8, Chen in view of Monteiro teaches the image processing method of claim 1, further comprising encoding the micro image (Monteiro, pg. 115733, section III.B.1, “Fig. 4 shows the conversion of the first 2×2 block when encoding the sixth SAI of the PVS from the example shown before in Fig. 3.”).
The rationale for obviousness is the same as provided for claim 1.
Regarding claim 9, Chen in view of Monteiro teaches the image processing method of claim 8, further comprising:
encoding a plurality of original main images corresponding to the plurality of main images (Chen, FIG. 2(a), “JEM Encoder”; The sampled SKVs are encoded. The decoded SKVs are decoded from the encoded sampled SKVs.), from among the plurality of light field images (from among all SKVs); and
transmitting information regarding the encoded micro image (output of the modified decoder) and information regarding the encoded original main images (Chen, FIG. 2, “compressed bit stream”; The compressed bit stream transmits the encoded main SKVs).
Regarding claim 10, Chen in view of Monteiro teaches the image processing method of claim 1, wherein the obtaining of the plurality of main images comprises:
encoding a plurality of original main images corresponding to the plurality of main images (Chen, FIG. 2(a), “JEM Encoder”; The sampled SKVs are encoded. The decoded SKVs are decoded from the encoded sampled SKVs.), from among the plurality of light field images (from among all SKVs); and
obtaining the plurality of main images by decoding the plurality of encoded original main images (Chen, FIG. 2(a), “Decoded SKV”).
Claims 11-14, 18, and 19 substantially correspond to claims 1-4, 8, and 9, respectively, by reciting an image processing apparatus comprising: a memory configured to store at least one instruction (See Chen at Fig. 11 and section 5.D; The SC-SKV codec produced compressed digital images, which requires the algorithms of the codec to at least be temporarily stored so that they can be used to process digital images); and at least one processor operating according to the at least one instruction (See Chen at Fig. 11 and section 5.D; The SC-SKV codec produced compressed digital images, which requires the algorithms of the codec to be executed by a computer (which includes a processor) to compress a digital video.), wherein the at least one processor is configured to perform the methods corresponding to claims 1-4, 8, and 9. The rationale for obviousness is the same for each corresponding claim.
Regarding claim 20, Chen in view of Monteiro teaches a computer-readable recording medium having recorded thereon a program for performing the method of claim 1 in a computer (See Chen at Fig. 11 and section 5.D; The SC-SKV codec produced compressed digital images, which requires the algorithms of the codec to at least be temporarily stored so that they can be executed by a computer to compress a digital video.).
Claims 5-7 and 15-17 are rejected under 35 U.S.C. 103 as being unpatentable over Chen in view of Monteiro and further in view of Variational Light Field Analysis for Disparity Estimation and Super-Resolution to Wanner et al. (hereinafter “Wanner”).
Regarding claim 5, Chen in view of Monteiro teaches the image processing method of claim 4, but does not teach that which is explicitly taught by Wanner.
Wanner teaches wherein the generating of the micro image by arranging the synthesized image by using the depth map comprises generating the micro image by adjusting a location of a pixel of the synthesized image based on a value of the depth map corresponding to the pixel of the synthesized image (Wanner, pg. 610, section 4.2, “After obtaining EPI disparity estimates … from the horizontal and vertical slices, respectively, we integrate those estimates into a consistent single disparity map … for each view”).
Chen in view of Monteiro is analogous to the claimed invention for the reasons provided above. Wanner discloses generating disparity maps by obtaining disparity estimates from slope lines in horizontal and vertical EPIs that correspond to unchanging intensity in the light field. Thus, Wanner shows that it was known in the art before the effective filing date of the claimed invention to generate disparity maps for LF data from EPIs, which is analogous to the claimed invention in that it is pertinent to the problem being solved by the claimed invention, improving light field image compression efficiency.
A person of ordinary skill in the art would have been motivated to replace the disparity quantization of the encoder as disclosed by Chen in view of Monteiro with the EPI-based disparity map generation disclosed by Wanner to thereby obtain horizontal and vertical epipolar plane images to generate a merged disparity map for a given view. Changes in disparity values in the disparity map cause pixels in the reconstructed image to be in different locations, thereby adjusting their locations. Based on the foregoing, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have made such modification according to known methods to yield the predictable results to have the benefit of maintaining global estimates of disparity for the non-sample SKVs.
Regarding claim 6, Chen in view of Monteiro teaches the image processing method of claim 4, but does not teach that which is explicitly taught by Wanner.
Wanner teaches wherein the generating of the micro image by arranging the synthesized image by using the depth map comprises performing a horizontal arrangement indicating an arrangement between pixels present in a same row of the synthesized image (Wanner, pg. 610, section 4.2, “After obtaining EPI disparity estimates … from the horizontal and vertical slices, respectively, we integrate those estimates into a consistent single disparity map … for each view”).
The rationale for obviousness is the same as provided for claim 5.
Regarding claim 7, Chen in view of Monteiro teaches the image processing method of claim 4, but does not teach that which is explicitly taught by Wanner.
Wanner teaches wherein the generating of the micro image by arranging the synthesized image by using the depth map comprises performing a vertical arrangement indicating an arrangement between pixels present in a same column of the synthesized image (Wanner, pg. 610, section 4.2, “After obtaining EPI disparity estimates … from the horizontal and vertical slices, respectively, we integrate those estimates into a consistent single disparity map … for each view”).
The rationale for obviousness is the same as provided for claim 5.
Claims 15-17 substantially correspond to claims 5-7, respectively, by reciting an image processing apparatus comprising: a memory configured to store at least one instruction (See Chen at Fig. 11 and section 5.D; The SC-SKV codec produced compressed digital images, which requires the algorithms of the codec to at least be temporarily stored so that they can be used to process digital images); and at least one processor operating according to the at least one instruction (See Chen at Fig. 11 and section 5.D; The SC-SKV codec produced compressed digital images, which requires the algorithms of the codec to be executed by a computer (which includes a processor) to compress a digital video.), wherein the at least one processor is configured to perform the methods corresponding to claims 5-7. The rationale for obviousness is the same for each corresponding claim.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to RYAN P POTTS whose telephone number is (571)272-6351. The examiner can normally be reached M-F, 9am-5pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sumati Lefkowitz can be reached at 571-272-3638. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RYAN P POTTS/Examiner, Art Unit 2672
/SUMATI LEFKOWITZ/Supervisory Patent Examiner, Art Unit 2672