Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claim 9 is objected to because of the following informalities:
Claim 9 recites: “A method comprising: capturing a video stream by a user equipment (UE); generating, by a semantic segmentation network in a processor of the UE, a first feature map based on a first frame of the video stream, wherein the first feature map comprises first information for generating a first segmentation and confidence map for the first frame; generating, by an infinite impulse response (IIR) filter of the processor, a corrected feature map based on the first feature map and corrected feature map information of a previous frame of the video stream; and generating, by the processor, the first segmentation and confidence map based on the corrected feature map.”
The language is confusing as it appears to suggest from the underlined above that the same “first segmentation” and “confidence map” generated based on “a/the first feature map” is also generated based on “the corrected feature map”.
Clarification is required.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-6, 16-19 are rejected under 35 U.S.C. 102(a)(1) as being anticipate by Xu et al, Dynamic Video Segmentation Network, IEEE, CVPR 2018.
Regarding claim 1,
A method comprising: capturing a video stream by a user equipment (UE) (Figs. 1-2; Note: UE for providing video stream could be autonomous vehicles, surveillance cameras, unmanned aerial vehicles (UAVs), etc., see Section 1: Introduction);
PNG
media_image1.png
580
766
media_image1.png
Greyscale
generating, by a semantic segmentation network in a processor of the UE, a first feature map based on a first frame of the video stream, wherein the first feature map comprises first information for generating a first segmentation (Fig. 2 above, “t” depicts “first frame” ) and confidence map for the first frame (Figs. 4, 6; page 6557, right col., last paragraph);
PNG
media_image2.png
290
746
media_image2.png
Greyscale
PNG
media_image3.png
306
724
media_image3.png
Greyscale
generating, by the processor, a second feature map for a second frame of the video stream based on the first feature map, wherein the second feature map comprises second information for generating a second segmentation and confidence map for the second frame (Fig. 2 above, “t+5” depicts “second frame”, clearly it’s based on first frame “t” i.e., similar scene at a later time; the second segmentation and confidence map for t-5 frame are similarly generated as “t” frame);
and generating, by the processor, the second segmentation and confidence map based on the second information (this is redundant from the immediately preceding claim language as underlined).
Regarding claim 2,
The method of claim 1, wherein generating the second feature map comprises: generating, by the processor, a first optical flow based on the first frame and the second frame; and warping, by the processor, the first feature map based on the first optical flow to generate the second feature map (Fig. 3; Section 3.1 “Dynamic Video Segmentation Network”).
PNG
media_image4.png
370
1218
media_image4.png
Greyscale
Regarding claim 3,
The method of claim 1, further comprising: generating, by the semantic segmentation network, a third feature map based on a third frame of the video stream, wherein the third feature map comprises third information for generating a third segmentation and confidence map for the third frame (See Fig. 4 above; based on the discussion in claims 1-2 above, a “third frame” is simply another keyframe and/or current frame in the video sequence, otherwise, feature mapping, segmentation and confidence mapping via confidence score are the same).
Regarding claim 4,
The method of claim 3, wherein generating the second feature map comprises: interpolating, by the processor, the first feature map and the third feature map to generate the second feature map (Pages 6559-61, Sections 3.1 and 3.4; Note: feature map prediction is interpolation).
PNG
media_image5.png
190
688
media_image5.png
Greyscale
PNG
media_image6.png
280
642
media_image6.png
Greyscale
PNG
media_image7.png
192
670
media_image7.png
Greyscale
Regarding claim 5,
The method of claim 3, wherein generating the second feature map comprises: generating, by the processor, a first optical flow based on the first frame and the second frame; warping, by the processor, the first feature map based on the first optical flow to generate a first warped feature map; generating, by the processor, a second optical flow based on the second frame and the third frame; warping, by the processor, the third feature map based on the second optical flow to generate a second warped feature map; and interpolating, by the processor, the first warped feature map and the second warped feature map to generate the second feature map (this is met by Fig. 3, “spatial warping path”; also Section 3.1).
PNG
media_image8.png
338
1146
media_image8.png
Greyscale
Regarding claim 6,
The method of claim 1, further comprising: generating, by the processor, an enhanced second frame by image signal processing the second frame based on the second segmentation and confidence map (Rejection of claims 1-2 is incorporated herein; also as depicted in Fig. 3 above and p.6557, right col., last par.) the expected confidence score dictates the decision network to opt for segmentation path or flow network path with spatial warping; the highest expected confidence score indicates the flow network to generate similar results as the segmentation network i.e., enhanced frame).
PNG
media_image9.png
254
514
media_image9.png
Greyscale
Regarding claim 16,
A user equipment (UE) comprising: a processor; and a non-transitory computer readable storage medium storing instructions that, when executed, cause the processor to: capture a video stream; generate, by a semantic segmentation network, a first feature map based on a first frame of the video stream, wherein the first feature map comprises first information for generating a first segmentation and confidence map for the first frame; generate a second feature map for a second frame of the video stream based on the first feature map, wherein the second feature map comprises second information for generating a second segmentation and confidence map for the second frame; and generate the second segmentation and confidence map based on the second information (the rejections of claim 1 applies herein) .
Regarding claim 17,
The UE of claim 16, wherein, in generating the second feature map, the instructions further cause the processor to: generate a first optical flow based on the first frame and the second frame; and warp the first feature map based on the first optical flow to generate the second feature map (the rejections of claims 1-2 apply herein).
Regarding claim 18,
The UE of claim 16, wherein: the instructions further cause the processor to generate, by the semantic segmentation network, a third feature map based on a third frame of the video stream, wherein the third feature map comprises third information for generating third segmentation and confidence maps for the third frame; and in generating the second feature map, the instructions further cause the processor to interpolate the first feature map and the third feature map to generate the second feature map (the rejections of claims 1-4 apply herein).
Regarding claim 19,
The UE of claim 16, wherein: the instructions further cause the processor to generate, by the semantic segmentation network, a third feature map based on a third frame of the video stream, wherein the third feature map comprises third information for generating a third segmentation and confidence map for the third frame; and in generating the second feature map, the instructions further cause the processor to: generate a first optical flow based on the first frame and the second frame; warp the first feature map based on the first optical flow to generate a first warped feature map; generate a second optical flow based on the second frame and the third frame; warp the third feature map based on the second optical flow to generate a second warped feature map; and interpolate the first warped feature map and the second warped feature map to generate the second feature map (the rejections of claims 1-5 apply herein).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 7-8, 9-15, 20 are rejected under 35 U.S.C. 103 as being unpatentable over Xu et al and further in view of US10547871B2 (Aydin et al).
Regarding claim 7,
The rejection of claim 1 is incorporated herein. Xu et al teaches enhancing a subsequent frame through the flow network via warping (Fig. 3 above). However, Xu et al does not teach the following limitations as further recited. Aydin et al teaches:
The method of claim 1, wherein generating the second segmentation and confidence map comprises: generating, by an infinite impulse response (IIR) filter of the processor, a corrected feature map based on the second feature map and information on the first feature map corrected by the IIR filter; and generating, by the processor, the second segmentation and confidence map based on the corrected feature map (Abstract; Fig. 1 and associated description e.g., pars. 0017, 0019-23; Note: IIR filtering enables “enhancement” of sparse optical into “dense” optical flow the input feature maps).
At the time of effective filing, it would have been obvious to incorporate the teaching of Aydin et al into Xu et al to utilize only one filter state of the IIR updated based on new video frames in the sequence for optimization purposes (Abstract; par. 0059).
Regarding claim 8,
Aydin et al further teaches, The method of claim 7, wherein the second segmentation and confidence map is generated by up-sampling the corrected feature map (par. 0059).
[0059] … As a result, techniques disclosed herein may be used in real time to perform filtering in many image and video domain applications such as optical flow estimation, HDR tone mapping, stylization, detail manipulation, sparse data upsampling, disparity estimation, visual saliency computation, and temporal consistency, among others.
Regarding claim 9,
A method comprising: capturing a video stream by a user equipment (UE); generating, by a semantic segmentation network in a processor of the UE, a first feature map based on a first frame of the video stream, wherein the first feature map comprises first information for generating a first segmentation and confidence map for the first frame; generating, by an infinite impulse response (IIR) filter of the processor, a corrected feature map based on the first feature map and corrected feature map information of a previous frame of the video stream; and generating, by the processor, the first segmentation and confidence map based on the corrected feature map (the rejections of claims 1 and 7 apply herein).
Regarding claim 10,
The method of claim 9, further comprising: generating, by the processor, an enhanced first frame by image signal processing the first frame based on the first segmentation and confidence map (the rejections of claims 1 and 7 apply herein).
Regarding claim 11,
The method of claim 9, wherein the first segmentation and confidence map is generated by up-sampling the corrected feature map (the rejections of claims 1, 7 and 8 apply herein).
Regarding claim 12,
The method of claim 9, further comprising: generating, by the processor, a first optical flow based on the first frame and a second frame of the video stream; and warping, by the processor, the first feature map based on the first optical flow to generate a second feature map for the second frame, wherein the second feature map comprises second information for generating a second segmentation and confidence map for the second frame (the rejections of claims 1-2 and 7 apply herein).
Regarding claim 13,
The method of claim 9, further comprising: generating, by the semantic segmentation network, a second feature map based on a second frame of the video stream, wherein the second feature map comprises second information for generating a second segmentation and confidence map for the second frame; and generating, by the processor, a third feature map for a third frame of the video stream based on the first feature map and the second feature map, wherein the third feature map comprises third information for generating a third segmentation and confidence map for the third frame (the rejections of claims 1-3 and 7 apply herein).
Regarding claim 14,
The method of claim 13, wherein generating the third feature map comprises interpolating the first feature map and the second feature map to generate the third feature map (the rejections of claims 1-4 and 7 apply herein).
Regarding claim 15,
The method of claim 13, wherein generating the third feature map comprises: generating, by the processor, a first optical flow based on the first frame and the third frame; warping, by the processor, the first feature map based on the first optical flow to generate a first warped feature map; generating, by the processor, a second optical flow based on the second frame and the third frame; warping, by the processor, the second feature map based on the second optical flow to generate a second warped feature map; and interpolating, by the processor, the first warped feature map and the second warped feature map to generate the a third feature map (the rejections of claims 1-5 and 7 apply herein).
Regarding claim 20,
The UE of claim 16, wherein the instructions further cause the processor to: generate, by an infinite impulse response (IIR) filter, a corrected feature map based on the second feature map and information on the first feature map corrected by the IIR filter, wherein the second segmentation and confidence map is generated by upscaling the corrected feature map; and generate an enhanced second frame by image signal processing the second frame based on the second segmentation and confidence map (the rejections of claims 1 and 7-8 apply herein)..
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to VU LE whose telephone number is (571)272-7332. The examiner can normally be reached M-F 8:00 - 17:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
Vu Le can be reached at 2-7332. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/VU LE/Supervisory Patent Examiner, Art Unit 2668
Prosecution Insights