Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 1/23/25 is being considered by the examiner.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory obviousness-type double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); and In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on a nonstatutory double patenting ground provided the conflicting application or patent either is shown to be commonly owned with this application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement.
Effective January 1, 1994, a registered attorney or agent of record may sign a terminal disclaimer. A terminal disclaimer signed by the assignee must fully comply with 37 CFR 3.73(b).
Claims 1, 13 and 20 are provisionally rejected on the ground of non-statutory obviousness-type double patenting as being unpatentable over claims 1 of US Patent No 12,243,170. Although the conflicting claims are not identical, they are not patentably distinct from each other because the present claims have the same scope of US Patent.
This is a provisional obviousness-type double patenting rejection because the conflicting claims have not in fact been patented.
Table 1 illustrates the conflicting claims.
Present Application 19/035,739
1, 13, 20
US Patent No 12,243,170
1
Table 2 provides a comparative mapping of the limitations of independent claims 1 of the present application when compared against the limitations of claims 1 of Patent Application No 12,243,170.
Present Application 19/035,739
US Patent No 12,243,170
1. A method comprising:
receiving a live camera feed of an object captured by a camera;
determining via a processor an augmented reality overlay data based on the object, the augmented reality overlay data include one or more tags, each of the tags characterizing a feature of the object, each of the tags being associated with a respective location on the object, each of the respective locations being represented in a reference view of the object, wherein a temporal filter is applied to overlay data locations by weighting or averaging displayed overlay data locations for a current frame based on overlay data locations determined in previous frames, wherein the temporal filter is used to generate a smooth visualization of a result; determining, for each of a plurality of frames in the live camera feed, a respective frame location for one or more of the tags, each of the respective frame locations determined based on a correspondence between the reference view of the object and the respective frame; and presenting the live camera feed on a display screen, the live camera feed including the plurality of frames, each of the plurality of frames including a respective one of the tags, each of the tags being positioned at the respective frame location.
A computing device comprising:
a camera configured to capture a live camera feed of an object;
a processor configured to: determine an object identity for an object represented the live camera feed during an initialization phase, the initialization phase including projecting one or more triangulated points into a designated frame based on camera pose information determined based on data collected from an inertial measurement unit at the computing device, wherein projecting the one or more triangulated points includes triangulating 2D skeleton joints into 3D skeleton joints, create an object model of the object by performing skeleton detection on the live camera feed;
determine augmented reality overlay data based on the object identity, the augmented reality overlay data including one or more tags, each of the tags characterizing a feature of the object, each of the tags being associated with a respective location on the object, each of the respective locations being represented in a reference view of the object, wherein a temporal filter is applied to the overlay data locations by weighting or averaging displayed overlay data locations for a current frame based on overlay data locations determined in previous frames, wherein the temporal filter is used to generate a smooth visualization of a result, and determine, for each of a plurality of frames in the live camera feed, a respective frame location for one or more of the tags, each of the respective frame locations determined based on a correspondence between the reference view of the object and the respective frame; and a display screen configured to present the live camera feed including the plurality of frames during a presentation phase after the initialization phase, each of the plurality of frames including a respective one of the tags, each of the tags being positioned at the respective frame location, wherein the initialization phase is performed again when the computing device determines that the live camera feed has changed more than a designated amount from the image present when initialization was last performed, wherein initialization comprises estimating a reference-to-live image correspondence using a surface mesh.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 8-14 and 18-20 are rejected under 35 U.S.C. 103 as being unpatentable over Holzer et al. (US 2017/0109930) in view of Beysserie et al. (US 2018/0041707).
As to Claim 1, Holzer teaches A method comprising:
receiving a live camera feed of an object captured by a camera (Holzer discloses “a live-action video or photo series…in which a simulated environment is depicted through video and/or image data” in [0002]);
determining via a processor an augmented reality overlay data based on the object, the augmented reality overlay data include one or more tags, each of the tags characterizing a feature of the object, each of the tags being associated with a respective location on the object, each of the respective locations being represented in a reference view of the object (Holzer discloses “receiving a selection of an anchor location in a reference image for a synthetic object to be placed within a multi-view image” in [0005]; “implementing augmented reality by adding a three-dimensional (3D) tag (also referred to herein as a synthetic object) such as an image, text, object, graphic, or the like to a multi-view image, where the multi-view image includes multiple views of a real-world environment” in [0021]; “a synthetic scene from the target view may be rendered using tracking information between the reference image and a target image from the multi-view image” in [0045]),
determining, for each of a plurality of frames in the live camera feed, a respective frame location for one or more of the tags, each of the respective frame locations determined based on a correspondence between the reference view of the object and the respective frame (Holzer discloses “augmented reality can take the form of a live-action video or photo series with added elements that are computer-generated” in [0002], see also [0020, 0027]; “a reference image 201 refers to a view (i.e. reference view) in the multi-view image where an anchor location 209 is selected for a synthetic object to be placed in the multi-view image….a target image 203 refers to a view (i.e. target view) in the multi-view image for which a synthetic image is generated. This synthetic image is then overlaid on the target image to yield an augmented reality version of the target image. By generating synthetic images for various target views and overlaying these synthetic images on the corresponding target images, an augmented reality version of the multi-view image can be generated” in [0033]); and
presenting the live camera feed on a display screen, the live camera feed including the plurality of frames, each of the plurality of frames including a respective one of the tags, each of the tags being positioned at the respective frame location (Holzer discloses “a synthetic object that is rendered into a scene can be represented by a video” in [0039]; “The synthetic image produced from this virtual view is then overlaid on the target image at 410, and blended to produce a new, augmented image from the target view… this process can be repeated for multiple views in the multi-view image to generate an augmented reality version of the multi-view image that appears to include the synthetic object” in [0052]; see also Fig 1-3.)
Holzer doesn’t explicitly teach temporal filter. The combination of Beysserie further teaches following limitations:
wherein a temporal filter is applied to overlay data locations by weighting or averaging displayed overlay data locations for a current frame based on overlay data locations determined in previous frames, wherein the temporal filter is used to generate a smooth visualization of a result (Beysserie discloses “Operation 200 may begin by stabilizing incoming image frames using a desired video stabilization technique, e.g., using a filter that combines, in some fashion, the motion data associated with a current incoming frame and the motion data associated with the previous frame, in order to compute smoothed stabilization motion adjustment values for the current frame… Next, for each "past" frame of the selected frame (moving backward in time----one frame at a time-in "reverse temporal" order) (block 206), the operation 200 may perform the following steps” in [0058]; “In other words, the filter will start with a "current" frame and then compute a smooth trajectory from the output of the previous frame in the time-sequenced set of images to the current frame based on a weighted combination of the current frame and the previous frame, e.g., a 20% contribution to the stabilization from the position of the current frame and an 80% contribution to the stabilization from the stabilized position of the previous frame” in [0060].)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the invention of Holzer with the teaching of Beysserie so as to stabilize incoming image frames using a desired video stabilization technique (Beysserie, [0058]).
As to Claim 2, Holzer in view of Beysserie teaches The method recited in claim 1, the method further comprising:
for each of the frames, determining the correspondence between the reference view of the object and the respective frame (Holzer discloses “a reference image 201 refers to a view (i.e. reference view) in the multi-view image where an anchor location 209 is selected for a synthetic object to be placed in the multi-view image…. a target image 203 refers to a view (i.e. target view) in the multi-view image for which a synthetic image is generated” in [0033]; see also Fig 2-3.)
As to Claim 8, Holzer in view of Beysserie teaches The method recited in claim 1, wherein the reference view of the object is a multi-view interactive digital media representation, the multi-view interactive digital media representation including a plurality of images of the object, each of the images of the object being captured from a different perspective view (Holzer discloses “a surround view is constructed from multiple images that are captured from different locations” in [0026]; “a surround view is a multi-view interactive digital media representation” in [0027].)
As to Claim 9, Holzer in view of Beysserie teaches The method recited in claim 8, wherein the multi-view interactive digital media representation is navigable in one or more dimensions (Holzer discloses the user can navigates through the multi-view image in [0039].)
As to Claim 10, Holzer in view of Beysserie teaches The method recited in claim 9, the method further comprising: generating the multi-view interactive digital media representation via the processor (Holzer discloses “computer processor is used to create a three-dimensional model that includes the content and context of the surround view” in [0026].)
As to Claim 11, Holzer in view of Beysserie teaches The method recited in claim 10, determining a three-dimensional model of the object based on the multi-view interactive digital media representation (Holzer discloses “computer processor is used to create a three-dimensional model that includes the content and context of the surround view” in [0026].)
As to Claim 12, Holzer in view of Beysserie teaches The method recited in claim 1, wherein the object is a vehicle, and wherein the reference view of the object includes each of a left vehicle door, a right vehicle door, and a windshield (Holzer discloses “a multi-view image data set shows a scene from different angles. For instance, a multi-view image data set can be captured while rotating a camera around its own center (panoramic case), while rotating the camera around one or multiple objects of interest (object case), while translating the camera, or while combining these movements” in [0028]; 3D model of the scene in [0053]. Official notice has been taken of the fact that “a scene of augment reality application may include an object like a vehicle to show any portion of the vehicle, which is well-known in the art (see MPEP 2144.03).)
Claim 13 recites similar limitations as claim 13 but in a system form. Therefore, the same rationale used for claim 13 is applied.
Claim 14 is rejected based upon similar rationale as Claim 2.
Claim 18 is rejected based upon similar rationale as Claim 8.
Claim 19 is rejected based upon similar rationale as Claims 9 & 10.
Claim 20 recites similar limitations as claim 13 but in a computer readable medium form. Therefore, the same rationale used for claim 13 is applied.
Claims 3-7 and 15-17 are rejected under 35 U.S.C. 103 as being unpatentable over Holzer in view of Beysserie and Floch (GB 2573170 A).
As to Claim 3, Holzer in view of Beysserie teaches The method recited in claim 1, wherein the live camera feed is divided into an initialization phase and a presentation phase, the initialization phase preceding the presentation phase (Holzer discloses “a synthetic object that is rendered into a scene can be represented by a video” in [0039]; camera calibration in [0051], see also Fig 1-3. It is well-known that a camera calibration may be performed before any operation. For example, Floch discloses “Preferably, the source cameras 12 are calibrated so that they output their source images of the scene at the same cadence and simultaneously. The intrinsic and extrinsic parameters of the cameras are supposed to be known or calculated by using well- known calibration procedures” in C5L19-22.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the invention of Holzer and Beysserie with the teaching of Floch so as to explain well-known device calibration.
As to Claim 4, Holzer in view of Beysserie and Floch teaches The method recited in claim 3, wherein the initialization phase involves projecting one or more triangulated points into a designated frame (Floch discloses “In some embodiments, the 2D-to-3D conversions of the pairs of matched 2D skeletons may involve triangulation, meaning generating a weak 3D skeleton from a pair of matched 2D skeletons includes: projecting a part of a first 2D skeleton of the pair as a first line in a 3D space; projecting the same part of the second 2D skeleton of the pair as a second line in the 3D space; and determining a 3D position locating the part for the weak 3D skeleton, based on the first and second lines” in C3L18-25; “Figure 8 schematically illustrates a triangulation way to build a weak 3D skeleton from a matching pair of two matched 2D skeletons according to embodiment of the present invention” in C4L25-27; “the source cameras 12 are calibrated so that they output their source images of the scene at the same cadence and simultaneously. The intrinsic and extrinsic parameters of the cameras are supposed to be known or calculated by using well- known calibration procedures. In particular, these calibration procedures allow the 3D object to be reconstructed into a 3D skeleton at the real scale” in C5L19-25; “Each pair of matching 2D skeletons from different views (source images) of the same scene volume can then be processed using triangulation in order to build an intermediate 3D skeleton, the robustness of which is quite low or weak. An intermediate or “weak” 3D skeleton can thus be generated, in 3D space, from each pair of matched 2D skeletons” in C7L10-13.)
As to Claim 5, Holzer in view of Beysserie and Floch teaches The method recited in claim 4, wherein the projection is performed based on camera pose information determined based on data collected from an inertial measurement unit associated with the camera (Holzer discloses “a scene which is captured as a multi-view image data set by a device that has an inertial measurement unit (IMU)… a multi-view image data set shows a scene from different angles… An IMU provides information about the orientation of a device while capturing the images” in [0028].)
As to Claim 6, Holzer in view of Beysserie and Floch teaches The method recited in claim 3, wherein the presentation phase involves triangulating a three-dimensional representation of the object for each of the frames (Holzer discloses a triangulated 3D multi-view representation in [0043]. Floch also discloses “In some embodiments, the 2D-to-3D conversions of the pairs of matched 2D skeletons may involve triangulation, meaning generating a weak 3D skeleton from a pair of matched 2D skeletons” in C3L18-20.)
As to Claim 7, Holzer in view of Beysserie and Floch teaches The method recited in claim 6, wherein the three-dimensional representation is triangulated based on the correspondence between the reference view and the respective frame (Holzer discloses “Accordingly, using the reference image, anchor location, and an estimate of the imaging device/camera's intrinsic parameters enables calculation of the synthetic image from other views” in [0044]; “a synthetic scene from the target view may be rendered using tracking information between the reference image and a target image from the multi-view image” in [0045]. Floch, C3L18-20.)
Claim 15 is rejected based upon similar rationale as Claim 3.
Claim 16 is rejected based upon similar rationale as Claims 4 & 5.
Claim 17 is rejected based upon similar rationale as Claims 6 & 7.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WEIMING HE whose telephone number is (571)270-1221. The examiner can normally be reached on Monday-Friday, 8:30am-5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Tammy Goddard can be reached on 571-272-7773. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see https://ppair-my.uspto.gov/pair/PrivatePair. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WEIMING HE/
Primary Examiner, Art Unit 2611
Prosecution Insights