DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Drawings
The drawings are objected to under 37 CFR 1.83(a). The drawings must show every feature of the invention specified in the claims. Therefore, the limitation “determining, by the computing system and based on the current location and the previously determined location, a stabilized location of the feature in the current frame by: selecting, by the computing system, an optimized pose for a virtual camera viewpoint in virtual space, wherein the optimized pose aligns the current location of the feature with the stabilized location of the feature, and warping the current frame so that the current frame appears to have been captured from the optimized pose of the virtual camera viewpoint rather than a pose of the physical camera” must be shown or the feature(s) canceled from the claim(s). No new matter should be entered.
Applicant’s Drawings show determining a stabilized location of the feature in the current frame occurs in Figure 3B, Box 340.
Applicant’s Drawings show selecting an optimized pose for a virtual camera viewpoint occurs in Figures 3C-3D, Box 360-380.
Applicant’s Drawings show warping the current frame so that the current frame appears to have been captured from the optimized pose of the virtual camera occurs in Figures 3E, Box 390 (Warping is discussed with reference to box 390 in paragraph 0081.).
Applicant’s Drawings show determining a stabilized location of the feature in the current frame occurs before (and is separate from) selecting an optimized pose for a virtual camera viewpoint and warping the current frame so that the current frame appears to have been captured from the optimized pose of the virtual camera. Therefore, Applicant’s Drawings do not show “determining…a stabilized location of the feature in the current frame by: selecting…an optimized pose for a virtual camera viewpoint in virtual space…, and warping the current frame so that the current frame appears to have been captured from the optimized pose of the virtual camera”.
Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. The figure or figure number of an amended drawing should not be labeled as “amended.” If a drawing figure is to be canceled, the appropriate figure must be removed from the replacement sheet, and where necessary, the remaining figures must be renumbered and appropriate changes made to the brief description of the several views of the drawings for consistency. Additional replacement sheets may be necessary to show the renumbering of the remaining figures. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 1-21 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
Claim 1 recites:
“determining, by the computing system and based on the current location and the previously determined location, a stabilized location of the feature in the current frame by:
selecting, by the computing system, an optimized pose for a virtual camera viewpoint in virtual space, wherein the optimized pose aligns the current location of the feature with the stabilized location of the feature, and
warping the current frame so that the current frame appears to have been captured from the optimized pose of the virtual camera viewpoint rather than a pose of the physical camera;”
However, applicant’s specification and drawings are not seen to disclose determining a stabilized location of the feature by: selecting an optimized pose for a virtual camera viewpoint in virtual space, and warping the current frame.
Applicant’s Specification, with reference to Applicant’s Drawings Figure 3B, recites:
[0048] At box 340, the computing system determines a stabilized location of the facial feature. Continuing the above-discussed example in which the face center the facial feature that is tracked, the computing system determines a stabilized center of the face (represented as location 182 in frame 140 of FIG. 1). The stabilized center of the face can represent a location at which the face center would be located had the face moved smoothly from the previous frame to the current frame. As such, the stabilized location of the face center may be selected to be not too far away from the real center of the face (represented as location 184 in frame 140 of FIG. 1), but the stabilized location of the face center is kept still if possible. As such, identifying the stabilized location of the face center can involve an optimization process that accounts for multiple different factors, as described with respect to boxes 342, 344, and 346.
[0049] At box 342, determining the stabilized location 182 of the facial feature accounts for the distance between a potential stabilized location and the actual location 184 of the facial feature (e.g., the mean of face landmarks). For example, the computing system may consider how far a potential stabilized face center strays from the actual, determined center of the face, to ensure that the stabilization does not attempt to stabilize the face so strongly that the video would depart substantially from the actual depiction of the face in the video. The term that accounts for this factor is referred to as E_follow, which measures how far away the stabilized two-dimensional center of the current frame H(T) is from the real landmark center C(T) that is determined as the mean of all 2D landmarks.
[0050] At box 344, determining the stabilized location 182 of the facial feature accounts for the distance between a potential stabilized location of the facial feature and a previously determined location 180 of the facial feature. For example, the computing system may consider how far a potential stabilized center strays from the location 180 of the last determined center of the face, to ensure that the stabilization does not attempt to track the current location of the face so strongly that the face makes sudden movements in the video. The term that accounts for this factor is referred to as E_smoothness, which measures the change between H(T) and H(T_pre), where H(T_pre) is the estimated 2D head center of the previous frame.
[0051] At box 346, determining the stabilized location of the facial feature accounts for a constraint on a distance between a potential stabilized location of the facial feature and the determined location of the facial feature. This factor is imposed as a hard constraint |H(T)−C(T)|<CroppedRange, so that C(T) moves within a valid range around H(T) which does not cause undefined regions.
As seen in Figure 3B, determining a stabilized location of the feature occurs in box 340. Box 340-346 do not include selecting an optimized pose for a virtual camera viewpoint in virtual space, and warping the current frame.
The limitation “selecting, by the computing system, an optimized pose for a virtual camera viewpoint in virtual space, wherein the optimized pose aligns the current location of the feature with the stabilized location of the feature” occurs in applicant’s Drawings Figure 3C-3D, Box 360-380.
The limitation “warping the current frame so that the current frame appears to have been captured from the optimized pose of the virtual camera viewpoint rather than a pose of the physical camera” occurs in applicant’s Drawings Figure 3E, Box 390. Applicant’s Specification recites: “[0081] At box 390, the computing system generates the stabilized view of the frame using the optimized pose of the virtual camera viewpoint. For instance, an image warping engine can load the mapping output from a motion filtering engine…”.
Applicant’s Specification teaches determining a stabilized location of the feature in the current frame occurs before (and is separate from) selecting an optimized pose for a virtual camera viewpoint and warping the current frame so that the current frame appears to have been captured from the optimized pose of the virtual camera. Therefore, Applicant’s Specification and Drawings do not teach “determining…a stabilized location of the feature in the current frame by: selecting…an optimized pose for a virtual camera viewpoint in virtual space…, and warping the current frame so that the current frame appears to have been captured from the optimized pose of the virtual camera”.
Claims 14 and 21 are rejected for the same reasons as claim 1.
Claims 2-13 and 15-20 are rejected as being dependent on claims 1 or 14.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-9, 14-19 and 21 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ryu et al. (US 2017/0078576 A1) in view of Bell et al. (US 2017/0332018 A1).
Regarding claim 14, Ryu et al. (hereafter referred as Ryu) teaches a computing system (Ryu, Figs. 1 and 3) comprising:
a camera (Ryu, Fig. 1, Camera 10, Paragraph 0039);
non-transitory computer-readable storage media (Ryu, Paragraph 0022) comprising computer-executable instructions that, when executed, cause one or more processors (Ryu, Paragraph 0051) of the computing system to perform operations comprising:
receiving, by the computing system, a video stream that includes multiple frames captured by a physical camera (Ryu, Paragraph 0043);
determining, by the computing system, a current location of a feature of an object in a current frame of the multiple frames of the video stream (Ryu, Fig. 7, location 731, Paragraphs 0073-0074 and 0106-0107);
determining, by the computing system, a previously determined location of the feature in a previous frame of the multiple frames of the video stream (Ryu, Fig. 7, location 711, Paragraphs 0073-0074 and 0106-0107);
determining, by the computing system and based on the current location and the previously determined location, a stabilized location of the feature in the current frame (Ryu, Fig. 5, Step 44, Paragraph 0091, Fig. 7, location 731 and location 721, Paragraph 0107-0108, A verified feature point is interpreted as a stabilized location since it is used for stabilizing the video.); and
generating, by the computing system, a stabilized view of the current frame using the stabilized location of the feature (Ryu, Fig. 5, Steps 44-47, Paragraph 0098).
However, Ryu does not teach selecting, by the computing system, an optimized pose for a virtual camera viewpoint in virtual space, wherein the optimized pose aligns the current location of the feature with the stabilized location of the feature, and
warping the current frame so that the current frame appears to have been captured from the optimized pose of the virtual camera viewpoint rather than a pose of the physical camera.
In reference to Bell et al. (hereafter referred as Bell), Bell teaches selecting, by the computing system, an optimized pose for a virtual camera viewpoint in virtual space, wherein the optimized pose aligns the current location of the camera with the stabilized location of the camera (Bell, Fig. 5, Step 540, Paragraphs 0047-0048 and 0057), and
warping the current frame so that the current frame appears to have been captured from the optimized pose of the virtual camera viewpoint rather than a pose of the physical camera (Bell, Fig. 5, Steps 550-590, Paragraphs 0035, 0047 and 0055-0059); and
generating, by the computing system, a stabilized view of the current frame using the stabilized location of the camera (Bell, Fig. 5, Paragraph 0060).
These arts are analogous since they are both related to imaging devices correcting for motion. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention (AIA ) to modify the invention of Ryu with the virtual camera determination as seen in Bell to provide for real-time video stabilization for mobile devices based on on-board motion sensing that do not alter an image when movement of the camera is below a movement threshold.
Claims 1 and 21 are rejected for the same reasons as claim 14.
Regarding claim 15, the combination of Ryu and Bell teaches the computing system of claim 14 (see claim 14 analysis), the operations further comprising: determining, by the computing system, differences between one or more candidate virtual poses of the virtual camera viewpoint and the pose of the physical camera, and wherein the selecting of the optimized pose comprises selecting a virtual camera pose from the one or more candidate virtual poses, and wherein the selecting of the virtual camera pose smoothens motion due to the virtual camera viewpoint with respect to the physical camera (Bell, Paragraphs 0047-0051, Hypothetical camera orientation/crop polygons are one or more candidate virtual poses.).
Claims 2 are rejected for the same reasons as claim 15.
Regarding claim 16, the combination of Ryu and Bell teaches the computing system of claim 14 (see claim 14 analysis), wherein the optimized pose has a different location and rotation in the virtual space than the pose of the physical camera (Bell, Paragraphs 0050, 0055 and 0057, The virtual camera pose is different from the physical camera pose. Rotation is also taken into account.).
Claims 3 are rejected for the same reasons as claim 16.
Regarding claim 17, the combination of Ryu and Bell teaches the computing system of claim 14 (see claim 14 analysis), wherein the selecting of the optimized pose accounts for a difference between a potential pose of the virtual camera viewpoint in the virtual space and a previous pose of the virtual camera viewpoint in the virtual space (Bell, Paragraphs 0047-0051).
Claims 4 are rejected for the same reasons as claim 17.
Regarding claim 18, the combination of Ryu and Bell teaches the computing system of claim 14 (see claim 14 analysis), the operations further comprising: determining, by the computing system and using information received from a movement or orientation sensor coupled to the physical camera, the pose of the physical camera in the virtual space (Ryu, Paragraphs 0072-0073, Bell, Fig. 100, motion sensing system 190, Paragraphs 0031, 0034 and 0057, “movement information”).
Claims 5 are rejected for the same reasons as claim 18.
Regarding claim 6, the combination of Ryu and Bell teaches the method of claim 5 (see claim 5 analysis), wherein the movement or orientation sensor comprises a gyroscope (Ryu, Paragraphs 0072-0073, Bell, Fig. 100, motion sensing system 190, Paragraphs 0031).
Regarding claim 19, the combination of Ryu and Bell teaches the computing system of claim 14 (see claim 14 analysis), wherein the operations for the determining of the stabilized location of the feature in the current frame comprise operations for determining that a distance between a potential stabilized location (Ryu, Fig. 7, locations 721) of the feature based on the identified location of the feature in the current frame Ryu, Fig. 7, location 731, Location 721 is a potential stabilized locations since it is within a certain distance from location 731.) of the feature in the current frame and the current location of the feature is within a cropped range of the video stream (Ryu, Fig. 7, predetermined distance 741, Paragraph 0106-0107, Predetermined distance 741 is interpreted as a cropped range.).
Claims 7 are rejected for the same reasons as claim 19.
Regarding claim 8, the combination of Ryu and Bell teaches the method of claim 7 (see claim 7 analysis), wherein the cropped range comprises defined regions of the current frame of the video stream around the current location of the feature (Ryu, Fig. 7, Crop range 741 defines a region around current feature location 731.).
Regarding claim 9, the combination of Ryu and Bell teaches the method of claim 7 (see claim 7 analysis), wherein the stabilized view of the current frame has a different cropping than a cropping of a stabilized view of the previous frame (Bell, Paragraph 0048, The virtual camera may move faster or slower. Moving the virtual camera will have a different cropping than a cropping of a stabilized view of the previous frame.).
Claim(s) 11-12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ryu et al. (US 2017/0078576 A1) in view of Bell et al. (US 2017/0332018 A1) in view of Tyagi et al. (US 2015/0362989 A1).
Regarding claim 11, the combination of Ryu and Bell teaches the method of claim 1 (see claim 1 analysis). However, the combination of Ryu and Bell does not teach wherein the determining of the current location of the feature of the object comprises: identifying a bounding box for the object; and identify, by identifying the location of multiple object landmarks within the bounding box that are depicted in the frame and determining a mean location of the multiple object landmarks, a center of the bounding box as the feature of the object.
In reference to Tyagi et al. (hereafter referred as Tyagi), Tyagi teaches identify a bounding box for the object; and identify, by identifying the location of multiple object landmarks within the bounding box that are depicted in the frame and determining a mean location of the multiple object landmarks, a center of the bounding box as the feature of the object (Tyagi, Figs. 2-3, bounding box 232/242, Paragraphs 0017 and 0020).
These arts are analogous since they are all related to tracking features in a video stream. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention (AIA ) to modify the combination of Ryu and Bell with the face detection method of Tyagi since to track facial features in the image.
Regarding claim 12, the combination of Ryu, Bell and Tyagi teaches the method of claim 11 (see claim 11 analysis). wherein the object is a human face, the feature is a center of the human face (Tyagi, Paragraph 0020), and wherein the multiple object landmarks comprise one or more of eye locations, ear locations, nose locations, eyebrow locations, mouth corner locations, or a chin location (Tyagi, Fig. 2, Paragraph 0017).
Claim(s) 13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ryu et al. (US 2017/0078576 A1) in view of Bell et al. (US 2017/0332018 A1) in further view of Yasuda (US 2010/0149369 A1).
Regarding claim 13, the combination of Ryu and Bell teaches the method of claim 1 (see claim 1 analysis). However, the combination of Ryu and Bell does not teach further comprising: selecting the object as a tracked object from among multiple objects depicted in the current frame by: selecting the tracked object based on sizes of each of the multiple objects, selecting the tracked object based on distances of each of the multiple objects to a center of the current frame, or selecting the tracked object based on distances between a tracked object selected in the previous frame and each of the multiple objects.
In reference to Yasuda, Yasuda teaches selecting the object as a tracked object from among multiple objects depicted in the current frame (Yasuda, Fig. 3, Paragraph 0062) by:
selecting the tracked object based on sizes of each of the multiple objects, (Yasuda, Fig. 3 and 6A, Paragraphs 0052-0053),
selecting the tracked object based on distances of each of the multiple objects to a center of the frame (Yasuda, Fig. 3 and 6B, Paragraphs 0052-0053), or
selecting the tracked object based on distances between a tracked object selected in a previous frame and each of the multiple objects (Yasuda, Fig. 2, Steps S206-S207, Paragraphs 0054-0056).
These arts are analogous since they are related to tracking positions in video streams. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention (AIA ) to modify the combination of Ryu and Bell with the method of determining a main face as seen in Yasuda to provide a technique in which when face detection is performed in successively captured images, each main face changeover to a face suitable for a main face is done while suppressing frequent main face changeovers (Yasuda, Paragraph 0024).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WESLEY JASON CHIU whose telephone number is (571)270-1312. The examiner can normally be reached Mon-Fri: 8am-4pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Twyler Haskins can be reached at (571) 272-7406. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WESLEY J CHIU/ Examiner, Art Unit 2639
/TWYLER L HASKINS/ Supervisory Patent Examiner, Art Unit 2639
Prosecution Insights