DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Interpretation
The specification uses the terms “flow map” and “optical flow map” interchangeably, so “flow map” is interpreted to mean “optical flow map”.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 1-20 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
The specification does not demonstrate that Applicant had possession of the full scope of the broad genus “map”. The claimed term “map” is a broad genus comprising many varied species including rigid transformations, affine transformations, perspective/projective transformations, deformation transformations, homographic transformations, motion disparity maps, depth change maps, optical flow maps, etc. The specification only discloses a single species of optical flow map. Thus, it does not disclose a representative number of species spanning the full scope of the genus “map”. Note that the varied species have substantial differences in how they are configured and implemented. Note that the specification uses the terms “flow map” and “optical flow map” interchangeably, so “flow map” is interpreted to mean “optical flow map”.
The specification does not demonstrate that Applicant had possession of the full scope of the broad genus “machine learning”. The claimed term “machine learning” is a broad genus comprising many varied species including support vector machines (SVM), random forests, decision trees, linear regression, K nearest neighbors, clustering, vanilla neural networks, transformers, generative adversarial networks, LSTM, RNN, convolutional neural network (CNN), etc. The specification only discloses a single species of CNNs. Thus, it does not disclose a representative number of species spanning the full scope of the genus “machine learning”. Note that the varied species have substantial differences in how they are trained, configured, and implemented.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, and 20 rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1, 1, 1, 2, 4, 5, 6, 7, 9, 10, 11, 11, 11, 12, 14, 15, 16, 17, 19, and 20 of U.S. Patent No. 11,989,938, respectively. Although the claims at issue are not identical, they are not patentably distinct from each other because limitations of the Application claims are present in the corresponding patent claims.
Claims 1, 8, 10, 11, 18, and 20 rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1, 2, 8, 13, 14, and 20 of U.S. Patent No. 11,676,381, respectively. Although the claims at issue are not identical, they are not patentably distinct from each other because limitations of the Application claims are present in the corresponding patent claims.
Claims 1, 8, 10, 11, and 20 rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1, 8, 3, 15, and 19 of U.S. Patent No. 10,929,673, respectively. Although the claims at issue are not identical, they are not patentably distinct from each other because limitations of the Application claims are present in the corresponding patent claims.
Claims 1, 8, 10, 11, 18, and 20 rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1, 1, 1, 12, 12, and 19 of U.S. Patent No. 10,474,900, respectively. Although the claims at issue are not identical, they are not patentably distinct from each other because limitations of the Application claims are present in the corresponding patent claims.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1, 8, 10, 11, 18, and 20 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by US 20130187952 A1 (Berkovich).
As per claim 1, Berkovich teaches a method comprising:
generating, using one or more processors of a device, a video sequence comprising a previous frame and a current frame (Berkovich:
para 38: “mobile device 10 includes a camera 16, a display 18 and at least one position sensor 20, all connected in data communication to a processing system 22 (including at least one processor 24 and data storage 26)”;
para 60: “Referring now to the client-side process of FIG. 2, a first image is obtained from the camera at step 42, and is uploaded to remote server 14”;
para 66: “the tracking process results in a transformation or mapping which allows pixel locations in a current image to be associated with corresponding locations in the first image and/or in an intermediate image. It should be noted in this context that the term "current image" is used to refer to an image sampled recently by the camera to form the basis for substantially real-time display of AR supplementary information.”;
para 71: “display the AR supplementary information in a continuously updating manner based on continuous video imagery sampled by the camera.”;
produce or capture a video that has at least two frames.
PNG
media_image1.png
1082
910
media_image1.png
Greyscale
);
detecting that the current frame does not have a corresponding modified current frame (Berkovich:
para 60: “The uploaded image will be processed by the server (to be described separately below) so as to achieve registration to a geographic database and provide data for correct alignment of AR information. However, the combined time of uploading the image and processing it at the server would introduce possibly large errors due to movement of the camera during the time lag.”;
para 61: “the processing system of the mobile unit performs image processing to track image motion between the first image and subsequent images obtained from the camera, and hence determines a mapping (step 46) which can be used to compensate for the delay in obtaining the results of the registration processing from the server”;
recognize that the current frame has not yet had modified. The client-side recognizes that the server-side has not yet supplied the modified current frame due to lag.
Para 72: “the mobile device preferably evaluates a tracking reliability indicator to assess when the reliability of the tracking has dropped (step 58), and then initiates a new upload-and-registration process (steps 42-54) as needed. The tracking reliability indicator may be based upon one or more parameters such as: elapsed time since sampling of the first image”: that is, the lag is too great since server-side has provided a modified current frame);
generating a map between the previous frame and the current frame (Berkovich:
abstract: “Image processing is then performed to track image motion between that image and subsequent images obtained from the camera, determining a mapping between the uploaded image and a current image.”;
para 8: “performing image processing to track image motion between the first image and subsequent images obtained from the camera, and hence determining a mapping between features of a reference image and features of a current image obtained from the camera, the reference image being selected from the group consisting of: the first image; and a second image for which transformation parameters were uploaded to the server corresponding to a mapping between the first image and the second image”;
“[0061] It is a particularly preferred feature of an aspect of the present invention that the processing system of the mobile unit performs image processing to track image motion between the first image and subsequent images obtained from the camera, and hence determines a mapping (step 46) which can be used to compensate for the delay in obtaining the results of the registration processing from the server.”;
“[0062] The phrase "tracking image motion" is used in this context to refer generically to a range of algorithms which allow association of pixels to pixels between spaced apart images in an image sequence. By way of specific non-limiting examples, the image motion tracking performed by mobile device 10 may be implemented according to any of at least three techniques”;
“[0064] A second implementation option employs optical flow.”;
Para 71 (shown below);
Fig. 2 (shown below): mainly 46:
PNG
media_image2.png
1287
1117
media_image2.png
Greyscale
create a transformation map (e.g. optical flow) indicating how pixels or features are moved from the previous frame to the current frame); and
generating the corresponding modified current frame by applying the map to a modified previous frame corresponding to the previous frame (Berkovich:
para 8: “employing the mapping to determine a corresponding pixel location for display of the supplementary information within the current image; and (f) displaying the supplementary information on the display correctly aligned with the view of the scene.”;
Fig. 2 (shown above): mainly 50-56;
PNG
media_image3.png
815
999
media_image3.png
Greyscale
PNG
media_image4.png
599
998
media_image4.png
Greyscale
warping the modified previous frame using the transformation map to generate a modified current frame).
As per claim 8, Berkovich teaches the method of claim 1, wherein the previous frame and the current frame are separated by a plurality of other frames in the video sequence (Berkovich: See arguments and citations offered in rejecting claim 1 above: especially paras 71 and 72 (both referenced above).
Para 66: “It should be noted in this context that the term "current image" is used to refer to an image sampled recently by the camera to form the basis for substantially real-time display of AR supplementary information. It should be noted that the "current image" may also be subject to processing delays of several frames, but is preferably still within the realm of what is considered acceptable as real time display, typically with a delay of significantly less than a second”;
The previous frame and current frame are not adjacent in frame sequence).
As per claim 10, Berkovich teaches the method of claim 1, further comprising:
displaying a modified video sequence on the device, the modified video sequence comprising the modified previous frame and the modified current frame,
wherein the modified video sequence collates the modified previous frame and the modified current frame (Berkovich: See arguments and citations offered in rejecting claim 1 above.
Figure 2 (shown above): mainly 70-74 – 50-56;
“[0069] At step 50, the mobile device 10 receives via the network data indicative of a pixel location for display of supplementary information within the first image, or an intermediate reference image. The mapping derived from the tracking process is then employed to determine a corresponding pixel location for display of the supplementary information within the current image (step 52). The term "pixel location" is used herein to refer to a location within an image defined relative to a grid of pixels. The location may be represented as a specific pixel address, as a group or region of pixels, or as some other geometrical relationship to one or more pixel location. The supplementary information is then displayed on the display correctly aligned with the view of the scene (step 54), either by displaying the supplementary information overlying an image of the view of the scene on a non-transparent display, or by displaying the supplementary information on a transparent display aligned with a direct view of the scene.”;
“[0071] Although the aforementioned process can optionally be performed with a number of still images, preferred functionality is to display the AR supplementary information in a continuously updating manner based on continuous video imagery sampled by the camera. In this case, the local tracking processing is preferably continued while the AR display is operating to generate updated mappings for new current images as they become available, and the pixel location for display of the supplementary information within the current image is updated according to the updated mapping (step 56). This effectively keeps the AR content correctly positioned within the moving video image or changing field of view of a direct-viewing device.”
stitch the machine learning modified frames together with the flow map warped frames to generate a final video that is displayed).
As per claim(s) 11 and 18, arguments made in rejecting claim(s) 1 and 8 are analogous, respectively. Berkovich also teaches A device comprising: one or more processors; a memory storing instructions that, when executed by the one or more processors, cause the device to perform operations comprising (Berkovich: See arguments and citations offered in rejecting claim 1 above;
Figs. 1, 3, 5; para 56: “software loaded into involatile memory and executed by the processing system”;
para 73, 88, 108: processor).
As per claim(s) 20, arguments made in rejecting claim(s) 1 are analogous. Berkovich also teaches a non-transitory machine-readable medium embodying instructions that, when executed by a machine, cause the machine to perform operations comprising (Berkovich: See arguments and citations offered in rejecting claim 1 above; Figs. 1, 3, 5; para 56: software).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 2-7, 9, 12-17, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Berkovich as applied to claims 1 and 11 above, and further in view of ZHU, XIZHOU, et al., "Deep Feature Flow for Video Recognition", ARXIV.ORG, CORNELL UNIVERSITY LIBRARY, 201 OLIN LIBRARY CORNELL UNIVERSITY ITHACA, NY 14853, (23 November 2016), 13 pgs (submitted in IDS) (Zhu).
As per claim 2, Berkovich teaches the method of claim 1, further comprising:
applying, using an editing engine, a
detecting a lag of the editing engine, wherein generating the map is in response to detecting the lag of the editing engine (Berkovich: See arguments and citations offered in rejecting claim 1 above:
“[0061] It is a particularly preferred feature of an aspect of the present invention that the processing system of the mobile unit performs image processing to track image motion between the first image and subsequent images obtained from the camera, and hence determines a mapping (step 46) which can be used to compensate for the delay in obtaining the results of the registration processing from the server.”;
Para 66: “It should be noted that the "current image" may also be subject to processing delays of several frames”;
Para 72: “mobile device preferably evaluates a tracking reliability indicator to assess when the reliability of the tracking has dropped (step 58), and then initiates a new upload-and-registration process (steps 42-54) as needed. The tracking reliability indicator may be based upon one or more parameters such as: elapsed time since sampling of the first image”;
generate the motion map when the scheme is lagging (e.g. relative to real-time)).
Berkovich does not teach machine learning.
Zhu teaches applying, using an editing engine, a machine learning scheme to the video sequence (Zhu:
PNG
media_image5.png
918
1093
media_image5.png
Greyscale
Introduction: page 1, column 2, para 2:
PNG
media_image6.png
1200
1090
media_image6.png
Greyscale
PNG
media_image7.png
935
1516
media_image7.png
Greyscale
PNG
media_image8.png
2293
1087
media_image8.png
Greyscale
Section 3: Page 4, col 2, paras 2-3:
PNG
media_image9.png
1308
1089
media_image9.png
Greyscale
PNG
media_image10.png
940
1085
media_image10.png
Greyscale
Note that the segmentation map and segmentation bounding boxes are modifications: See Figures 2 (shown above), 4, 5 (shown below):
PNG
media_image11.png
1438
1136
media_image11.png
Greyscale
PNG
media_image12.png
1419
1140
media_image12.png
Greyscale
).
Thus, it would have been obvious for one of ordinary skill in the art, prior to filing, to implement the teachings of Zhu into Berkovich since both Berkovich and Zhu suggest a practical solution and field of endeavor of using flow to efficiently propagate modifications pertaining to some frames to other frames in general and Zhu additionally provides teachings that can be incorporated into Berkovich in that the initial modifications are achieved via machine learning since “Deep convolutional neutral networks have achieved great success on image recognition tasks” (Zhu: abstract). The teachings of Zhu can be incorporated into Berkovich in that the initial modifications are achieved via machine learning. Furthermore, one of ordinary skill in the art could have combined the elements as claimed by known methods and, in combination, each component functions the same as it does separately. One of ordinary skill in the art would have recognized that the results of the combination would be predictable.
As per claim 3, Berkovich in view of Zhu teaches the method of claim 2, wherein detecting the lag of the editing engine is based on detecting that the current frame does not have the corresponding modified current frame (Berkovich: See arguments and citations offered in rejecting claim 2 above: especially para 72;
Lagging means that the editing engine has not yet modified the current frame).
As per claim 4, Berkovich in view of Zhu teaches the method of claim 2, wherein detecting the lag of the editing engine comprises:
determining that the editing engine is still processing images preceding the current frame, where the current frame is a next image to be processed (Berkovich: See arguments and citations offered in rejecting claim 2 above: especially para 72;
lagging means that the editing engine is still processing earlier frames).
As per claim 5, Berkovich in view of Zhu teaches the method of claim 2, further comprising:
generating modified images in a first pipeline by applying the machine learning scheme to the video sequence, the first pipeline comprising the previous frame having the corresponding modified previous frame and the current frame that does not have the corresponding modified current frame in the first pipeline; and
in response to detecting the lag, generating, in a second pipeline, the map between the previous frame and the current frame (Berkovich in view of Zhu: See arguments and citations offered in rejecting claim 2 above;
PNG
media_image13.png
467
999
media_image13.png
Greyscale
Also see paras 61 and 72 (both referenced above).
there are two pipelines. One pipeline pertains to the server-side editing engine. The other pipeline pertains to generating the client-side mapping).
As per claim 6, Berkovich in view of Zhu teaches the method of claim 5, wherein the second pipeline is asynchronous to the first pipeline (Berkovich: See arguments and citations offered in rejecting claim 5 above.
Note that para [0074] of the instant Application’s filed specification states that “editing engine 610 and the tracking engine 615 operate in different pipelines asynchronously (e.g., concurrently, in parallel).”: Thus, “asynchronously” can be interpreted as meaning “concurrently” or “in parallel”.
More broadly, “asynchronously” can be taken to mean: tasks occur independently).
As per claim 7, Berkovich in view of Zhu teaches the method of claim 5, wherein the first pipeline and the second pipeline are implemented on different threads of the one or more processors of the device (Berkovich: See arguments and citations offered in rejecting claim 5 above.
Each pipeline runs on its own thread).
As per claim 9, Berkovich in view of Zhu teaches the method of claim 2, wherein the machine learning scheme is trained to apply an image manipulation, and wherein the modified previous frame exhibits the image manipulation, wherein the map is a flow map that describes changes of image features in the video sequence (Berkovich in view of Zhu: See arguments and citations offered in rejecting claim 2 above.
Paras 10, 64: optical flow;
“[0062] The phrase "tracking image motion" is used in this context to refer generically to a range of algorithms which allow association of pixels to pixels between spaced apart images in an image sequence.”;
Fig. 2 (shown above): mainly 70-74 – 50-56;
the machine learning scheme alters an image. The map is an (e.g. optical) flow map that indicates how features move).
As per claim(s) 12-17 and 19, arguments made in rejecting claim(s) 2-7 and 9 are analogous, respectively. Berkovich also teaches A device comprising: one or more processors; a memory storing instructions that, when executed by the one or more processors, cause the device to perform operations comprising (Berkovich: See arguments and citations offered in rejecting claim 1 above;
Figs. 1, 3, 5; para 56: “software loaded into involatile memory and executed by the processing system”;
para 73, 88, 108: processor).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Zhu, as cited above in rejecting claim 2, also teaches all of the limitations of the claim 1.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Atiba Fitzpatrick whose telephone number is (571) 270-5255. The examiner can normally be reached on M-F 10:00am-6pm.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Bee can be reached on (571) 270-5183. The fax phone number for Atiba Fitzpatrick is (571) 270-6255.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
Atiba Fitzpatrick
/ATIBA O FITZPATRICK/
Primary Examiner, Art Unit 2677
Prosecution Insights