DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
The Amendment filed 07/29/2026 overcomes the following:
Objection to the title of the Specification;
Objection to claims 8 and 20 for informalities;
Interpretation of claim limitations in claims 9-12 under 35 USC 112(f); and
Rejection of claim 4 under 35 USC 112(b).
Applicant’s arguments with respect to claims 1-20 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument. The new grounds of rejection set forth below were necessitated by the amendment.
Claim Objections
Claims 10-12 are objected to because of the following informalities:
In claims 10-12, Examiner suggests correction of “wherein the processor is further to” to read as “wherein the processor is further configured to”.
Appropriate correction is required.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-6 and 9-18 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by US PG PUB. 2016/0198097 A1 (hereinafter “Yewdall”).
Regarding claim 1, Yewdall discloses a processing method (Yewdall, ¶0042-0048, Figs. 3A-B, 4A-E) comprising:
obtaining a frame of an original image in a spatial environment (Yewdall, ¶0042; “FIG. 3A shows an exemplary image frame 310 provided by the content source 110 of FIG. 2.”);
based on depth information of imaging objects in the original image, associating the imaging objects with a plurality of layers of the original image, the plurality of layers having a difference in a depth direction, the associating the imaging objects with the plurality of layers including: determining depth ranges of the imaging objects in the depth direction based on the depth information; determining, according to the depth ranges of the imaging objects, depth sections usable to perform layer division, wherein each of the depth sections at least includes a depth range corresponding to an entire imaging object of the imaging objects in the depth direction; and dividing image information of each of the depth sections into one layer to obtain the plurality of layers, such that entire image information of each of the imaging objects is divided into a same layer of the plurality of layers (Yewdall, ¶0042-0044; “The depth extraction circuit 210 and the depth-layering circuit 220 may receive the content source 110, and extract and separate the image frame 310 into multiple depth- layers 320, 330, and 340…The processing circuit 130 may create the depth- layers 320, 330, and 340 using techniques for converting 2D images into stereoscopic 3D images or through the use of image segmentation tools Image segmentation tools generally group neighboring pixels with similar characteristics in segments or superpixels. These image segments may represent parts of meaningful objects that can be used to make inferences about the contents of the image…the depth- layers 320, 330, and 340 are described or positioned as a back layer 320, a middle layer 330, and a front layer 340. The back layer 320 contains a mountain terrain, the middle layer 330 contains trees, and the front layer 340 contains a car. As described in FIG. 2, the depth layering circuit 220 may send the depth- layers 320, 330, and 340 to the motion tracking circuit 230, and the motion tracking circuit 230 may send the depth- layers 320, 330, and 340 to the compositing/editing circuit 250. According to another embodiment, the compositing/editing circuit 250 uses the depth- layers 320, 330, and 340 to sort pixels within the image frame 310 into different depth ranges. The compositing/editing circuit 250 assigns each pixel in the image frame 310 to fall within a pixel in one of the depth- layers 320, 330, and 340. The pixels are assigned to create the desired separation of objects within the image frame 310. Accordingly, each assigned pixel in the depth- layers 320, 330, and 340 may be found in the image frame 310.”).
performing corresponding image processing on a target object in a target layer according to the depth information to obtain a frame of a target image (Yewdall, Fig. 4A- 4E, ¶0046-0048; “FIG. 4C shows an exemplary final image 430 created by compositing, by the compositing/editing circuit 250, the object image 410 with the insert layer 412 located in front of the front layer 340. Accordingly, the person 420 of the object image 410 is in front of the car of the front layer 340, the trees of the middle layer 330, and the mountain terrain of the back layer 320.”).
Regarding claim 2, claim 1 is incorporated, and Yewdall further discloses wherein based on the depth information of the imaging objects in the original image, associating the imaging objects with the plurality of layers of the original image further includes:
obtaining the depth information of the imaging objects of the original image through a sensor (Yewdall, ¶0043; “The processing circuit 130 may further be configured to acquire or generate depth information for generating the depth- layers 320, 330, and 340 using a number of different techniques, for example, time-of-flight cameras, structured-light systems and depth-from-stereo hardware”);
Regarding claim 3, claim 1 is incorporated, and Yewdall further discloses wherein based on the depth information of the imaging objects in the original image, associating the imaging objects with the plurality of layers of the original image includes at least one of:
dividing the original image into the plurality of layers based on moving information and the depth information of the imaging objects to associate the imaging objects with the corresponding layers, the moving information representing moving statuses of the imaging objects (Yewdall, ¶0043-0044; ““The content source 110 may provide the image frame 310 with insertion positions marked by metadata or may include metadata for motion tracking provided by the metadata extraction circuit 260. Other circuit compositions may in turn use the metadata to identify the different depth-layers 320, 330, and 340 …The processing circuit 130 may create the depth- layers 320, 330, and 340 using techniques for converting 2D images into stereoscopic 3D images or through the use of image segmentation tools Image segmentation tools generally group neighboring pixels with similar characteristics in segments or superpixels. These image segments may represent parts of meaningful objects that can be used to make inferences about the contents of the image.”);
dividing the original image into the plurality of layers based on position areas and the depth information of the imaging objects in the original image to associate the imaging objects with the corresponding layers (Yewdall, ¶0043-0044; “The compositing/editing circuit 250 assigns each pixel in the image frame 310 to fall within a pixel in one of the depth- layers 320, 330, and 340. The pixels are assigned to create the desired separation of objects within the image frame 310. Accordingly, each assigned pixel in the depth- layers 320, 330, and 340 may be found in the image frame 310.”); or
dividing the original image into the plurality of layers based on the moving information, the position areas, and the depth information to associate the imaging objects with the corresponding layers (Yewdall, ¶0043-0044; “The content source 110 may provide the image frame 310 with insertion positions marked by metadata or may include metadata for motion tracking provided by the metadata extraction circuit 260. Other circuit compositions may in turn use the metadata to identify the different depth- layers 320, 330, and 340 …The processing circuit 130 may create the depth- layers 320, 330, and 340 using techniques for converting 2D images into stereoscopic 3D images or through the use of image segmentation tools Image segmentation tools generally group neighboring pixels with similar characteristics in segments or superpixels. These image segments may represent parts of meaningful objects that can be used to make inferences about the contents of the image.”).
Regarding claim 4, claim 3 is incorporated, and Yewdall further discloses wherein dividing the original image into the plurality of layers based on the moving information and the depth information of the imaging objects includes:
determining contour information of the imaging objects based on the moving information and an object recognition algorithm (Yewdall, ¶0043-0044; “The content source 110 may provide the image frame 310 with insertion positions marked by metadata or may include metadata for motion tracking provided by the metadata extraction circuit 260. Other circuit compositions may in turn use the metadata to identify the different depth- layers 320, 330, and 340 …The processing circuit 130 may create the depth- layers 320, 330, and 340 using techniques for converting 2D images into stereoscopic 3D images or through the use of image segmentation tools Image segmentation tools generally group neighboring pixels with similar characteristics in segments or superpixels. These image segments may represent parts of meaningful objects that can be used to make inferences about the contents of the image.”);
determining depth ranges of the imaging objects based on the contour information and the corresponding depth information (Yewdall, ¶0043-0044; “the compositing/editing circuit 250 uses the depth- layers 320, 330, and 340 to sort pixels within the image frame 310 into different depth ranges”); and
dividing the original image into the plurality of layers based on the depth ranges of the imaging objects (Yewdall, ¶0043-0044; “The compositing/editing circuit 250 assigns each pixel in the image frame 310 to fall within a pixel in one of the depth- layers 320, 330, and 340. The pixels are assigned to create the desired separation of objects within the image frame 310”).
Regarding claim 5, claim 3 is incorporated, and Yewdall further discloses wherein dividing the original image into the plurality of layers based on the position areas and the depth information includes:
determining boundary points of the imaging objects based on the position areas (Yewdall, ¶0043-0044; “The processing circuit 130 may create the depth- layers 320, 330, and 340 using techniques for converting 2D images into stereoscopic 3D images or through the use of image segmentation tools Image segmentation tools generally group neighboring pixels with similar characteristics in segments or superpixels. These image segments may represent parts of meaningful objects that can be used to make inferences about the contents of the image.”);
determining depth ranges of the imaging objects corresponding to the boundary points; and dividing the original image into the plurality of layers based on the depth ranges of the imaging objects (Yewdall, ¶0044; “the depth- layers 320, 330, and 340 are described or positioned as a back layer 320, a middle layer 330, and a front layer 340. The back layer 320 contains a mountain terrain, the middle layer 330 contains trees, and the front layer 340 contains a car…the compositing/editing circuit 250 uses the depth- layers 320, 330, and 340 to sort pixels within the image frame 310 into different depth ranges. The compositing/editing circuit 250 assigns each pixel in the image frame 310 to fall within a pixel in one of the depth- layers 320, 330, and 340. The pixels are assigned to create the desired separation of objects within the image frame 310.”).
Regarding claim 6, claim 1 is incorporated, and Yewdall further discloses wherein performing the corresponding image processing on the target object at the target layer according to the depth information to obtain the frame of the target image includes:
determining attribute information of the target object, and determining a corresponding image processing strategy based on the attribute information; determining a to-be-processed area of the target object in the corresponding target layer based on the depth information to perform processing in the to-be-processed area according to the corresponding image processing strategy (Yewdall, ¶0046-0048; “One way of inserting the insert layer 412 in front of the front layer 340 is to replace pixel values of the front layer 340, the middle layer 330, and the back layer 320 with overlapping pixels of the person 420 in the insert layer 412. The pixels in the front layer 340, the middle layer 330, and the back layer 320 that are not overlapping with the pixels of the person 420 in the insert layer 412 may remain intact.” The positions of the overlapping pixels of each object in each layer of the divided image are read here as the determined “attribute information” which is subsequently used to decide whether or not a pixel should be replaced with the object image of a desired insert object layer); and
editing a processing result to obtain the frame of the target image (Yewdall, Fig. 4C, ¶0046-0048; “FIG. 4C shows an exemplary final image 430 created by compositing, by the compositing/editing circuit 250, the object image 410 with the insert layer 412 located in front of the front layer 340. Accordingly, the person 420 of the object image 410 is in front of the car of the front layer 340, the trees of the middle layer 330, and the mountain terrain of the back layer 320.”);
wherein: one or more target layers or one or more target objects are provided; and the one or more target layers are in a one-to-one correspondence with the one or more target objects (Yewdall, ¶0043-0048; “the person 420 of the object image 410 is in front of the car of the front layer 340, the trees of the middle layer 330, and the mountain terrain of the back layer 320.”).
Claim 9 recites an apparatus having features which correspond directly to the elements of method claim 1, the rejection of which is applicable here, and Yewdall further discloses a camera, a processor and a memory (Yewdall, Fig. 1, ¶0026-0031).
Claim 10 recites an apparatus having features which correspond directly to the elements of method claim 2, the rejection of which is applicable here.
Claim 11 recites an apparatus having features which correspond directly to the elements of method claim 3, the rejection of which is applicable here.
Claim 12 recites an apparatus having features which correspond directly to the elements of method claim 4, the rejection of which is applicable here.
Claim 13 recites a device having features which correspond directly to the elements of method claim 1, the rejection of which is applicable here, and Yewdall further discloses a device comprising one or more processors (Yewdall, ¶0026).
Claim 14 recites a device having features which correspond directly to the elements of method claim 2, the rejection of which is applicable here.
Claim 15 recites a device having features which correspond directly to the elements of method claim 3, the rejection of which is applicable here.
Claim 16 recites a device having features which correspond directly to the elements of method claim 4, the rejection of which is applicable here.
Claim 17 recites a device having features which correspond directly to the elements of method claim 5, the rejection of which is applicable here.
Claim 18 recites a device having features which correspond directly to the elements of method claim 6, the rejection of which is applicable here.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 7-8 and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Yewdall, as applied to claims 1 and 13 above, in view of US PG PUB. 2015/0091900 A1 (hereinafter “Yang”).
Regarding claim 7, claim 6 is incorporated, and Yewdall does not expressly teach the limitations as further claimed, but, in an analogous field of endeavor, Yang does as follows.
Yang teaches wherein performing the corresponding image processing on the target object at the target layer according to the depth information includes: performing trapezoidal correction on a first target object in a first target layer according to depth information of the first target object (Fig. 3, ¶0037; “In order to perform perspective correction, the depth map is used to separate (304) the face image data from background image data. The term face image data is used here to indicate pixels from the original image data corresponding to a face identified from within the image. The background image data refers to pixels that are not part of the face image data. A warp (306) can be applied to the face image data based upon the depth map to rerender the face from a more distant viewpoint, thereby correcting for perspective distortion.”).
Yang is considered analogous art because it pertains to image processing based on depth information. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method taught by Yewdall to include perspective correction of target objects based on depth information, as taught by Yang, in order to remove perspective distortion in the image acquired by the camera (Yang, ¶0045).
Regarding claim 8, claim 7 is incorporated, and Yewdall further teaches wherein performing the image processing on the target object at the target layer according to the depth information includes at least one of:
erasing a second target object at a second target layer according to depth information of the second target object; or erasing or blurring a third target object at a third target layer according to depth information of the third target object (Yewdall, ¶0046-0048; “One way of inserting the insert layer 412 in front of the front layer 340 is to replace pixel values of the front layer 340, the middle layer 330, and the back layer 320 with overlapping pixels of the person 420 in the insert layer 412. The pixels in the front layer 340, the middle layer 330, and the back layer 320 that are not overlapping with the pixels of the person 420 in the insert layer 412 may remain intact.” Replacing the pixel values of certain parts of each of the layers with an inserted object is read here as “erasing” the existing object in that corresponding layer.).
Claim 19 recites a device having features which correspond directly to the elements of method claim 7, the rejection of which is applicable here.
Claim 20 recites a device having features which correspond directly to the elements of method claim 8, the rejection of which is applicable here.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. The additionally cited references pertain to image separation into depth layers and layer-specific image processing.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SAMAH A BEG whose telephone number is (571)270-7912. The examiner can normally be reached M-F 9 AM - 5 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, HENOK SHIFERAW can be reached at 571-272-4637. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SAMAH A BEG/Primary Examiner, Art Unit 2676
Prosecution Insights