DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments with respect to claim(s) 10-21 have been considered but are not persuasive.
The Double Patenting Rejection is maintained. After this Office Action, It will no longer be held in abeyance. See MEP 804 (“As filing a terminal disclaimer, or filing a showing that the claims subject to the rejection are patentably distinct from the reference application’s claims, is necessary for further consideration of the rejection of the claims, such a filing should not be held in abeyance. Only compliance with objections or requirements as to form not necessary for further consideration of the claims may be held in abeyance until allowable subject matter is indicated. Replies with an omission should be treated as provided in MPEP § 714.03.”)
Applicant argues
PNG
media_image1.png
172
574
media_image1.png
Greyscale
Examiner’s response:
The claim does not recite that the plurality of stored maps are pre-existing maps of the entire environment (as opposed to specific objects in the environment). A stored map of a specific object within the environment would read on the claim limitation “stored maps of the environment”. A stored map is not the same thing as a “3-d visual representation of the environment”, which according to the claim is based on a plurality of stored maps (i.e. objects which are updated using 2-d features)
Applicant argues :
PNG
media_image2.png
222
562
media_image2.png
Greyscale
Examiner’s response:
Lee discloses determining the pose from different sensors for different rates (faster, slower and same rate as camera). See for example paragraph 52. Therefore the first rate could be any speed, based on the type of the sensor.
Paragraph 70 states “To illustrate, the image alignment and AR processes may update/cycle at the same frame rate as the imaging cameras 114, 116, and 118 because these processes are directly tied to the captured imagery. “. Thus the updating of the environment happens at the same rate as the image cameras. Thus the second rate is faster than some first rates and slower than some first rates and equal to some first rates.
Paragraph 70 further states “Conversely, the location-related features of the electronic device 100 may not require a high position resolution, and thus the image analysis process to determine the current position/orientation of the electronic device 100 may occur at a cycle rate slower than the frame rate of the imaging cameras.”. Thus the second rate could be slower the rate of the image cameras. This still clearly anticipates “at a rate slower than the first rate”.
In response to Applicants argument to claim 13 and 19, the head is treated as it is in a fixed spatial position. Since the head is in a fixed position all previously estimated positions are also fixed.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP §§ 706.02(l)(1) - 706.02(l)(3) for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/process/file/efs/guidance/eTD-info-I.jsp.
Claim 10 and 16 are rejected on the ground of nonstatutory double patenting as being unpatentable over claim 3 of U.S. Patent No. 11,017,610 B2 (’610 patent)
Although the claims at issue are not identical, they are not patentably distinct from each other because of the following reason.
Claim 1
Claim 3 of the ’610 patent
A method, comprising:
tracking, at an electronic device, motion of the electronic device relative to an environment of the electronic device based on feature descriptors of one or more spatial features representing one or more objects in the environment of the electronic device, wherein the feature descriptors are generated based on images captured from one or more visual sensors and non-visual data from one or more non-image sensors at a first rate;
updating, at a second rate slower than the first rate, a three-dimensional visual representation of the environment that is based on a plurality of stored maps of the environment that were previously generated by the electronic device during prior mapping sessions, wherein the plurality of stored maps comprise stored feature descriptors based on periodic inputs of the generated feature descriptors and identify previously identified spatial features and their corresponding poses;
matching the one or more spatial features to spatial features of the plurality of stored maps to generate a first localized pose of the electronic device.
Claim 1. A method, comprising:
modifying, at an electronic device, a three-dimensional representation of an environment of the electronic device based on known feature descriptors of a stored plurality of maps of the environment that were previously generated by the electronic device during prior mapping sessions and a first map file comprising accumulated first mapping data received from a motion tracking module, wherein the first mapping data is based on images captured from one or more visual sensors and non-visual data from one or more non-image sensors;
in response to receiving a second map file comprising accumulated second mapping data from the motion tracking module after receiving an indication from the motion tracking module that a fault condition exists, wherein the fault condition comprises the electronic device changing direction at a rate that exceeds a rate at which the motion tracking module can generate accurate mapping data, buffering or discarding the second map file and omitting the second map file from the three-dimensional representation . . .
2. The method of claim 1, wherein the first, second, and third mapping data comprise feature descriptors of one or more spatial features representing an object in the environment of the electronic device, and wherein modifying the three-dimensional representation based on the third map file comprises matching feature descriptors of the third mapping data to feature descriptors of the first mapping data.
3. The method of claim 2, further comprising building a bundle adjustment problem comprising the feature descriptors of the three-dimensional representation and solving the bundle adjustment problem to generate a localized pose of the electronic device.
b. Regarding claim 16, claim 16 is analogous and corresponds to claim 10. See rejection of claim 10 for further explanation.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 10-14 and 16-20 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Lee (US 2015/0071524 A1).
Regarding claim 10, Lee discloses a method, comprising:
tracking, at an electronic device, motion of the electronic device relative to an environment of the electronic device based on generated feature descriptors of one or more spatial features representing one or more objects in the environment of the electronic device (Lee discloses that “the electronic device 100 estimates its current pose in response to, for example, a desire to verify its relative or absolute location in the local environment or to identify an object present in the local environment” at Fig. 1-100, Fig. 9-1000, Fig. 13-1302, ¶ 0114), wherein the generated feature descriptors are generated based on images captured from one or more visual sensors and non-visual data from one or more non-image sensors at a first rate (Lee discloses that “the current (camera) pose, includes the observation direction of one or more imaging cameras, and further can include one or more position-based parameters (e.g., orientation, 3D position, elevation, distance from an object or reference point), one or more non-position context parameters (e.g., time of day, ambient light), or a combination thereof. These parameters can be determined from the set 810 of non-image sensors, determined from reference features present in one or more captured images, and the like” at Fig. 8-113, 118, 116, 810, Fig. 13, ¶ 0114. Moreover, Lee discloses that “different sensors may be read at different rates or frequencies. For example, an ambient light reading may be taken only once for every N image captures by the imaging cameras 114, 116, and 118, whereas a six-degrees-of-freedom (6DoF) reading from the gyroscope may be taken every image capture so as to enable detection of the relative orientation of the electronic device 100 when the corresponding image was captured. Still further, accelerometer readings may be obtained at a rate much higher than the image capture rate so as to facilitate a more accurate internal navigation determination by the electronic device” at ¶ 0052) (emphasis added);
updating, at a second rate slower than the first rate, a three-dimensional visual representation of the environment that is based on a plurality of stored maps of the environment that were previously generated by the electronic device during prior mapping sessions, wherein the plurality of stored maps comprise stored feature descriptors based on periodic inputs of the generated feature descriptors and identify previously identified spatial features and their corresponding poses; (Lee, “[0098] The 3D feature descriptor 1010, after initialized with one or more 2D feature descriptors, may be subsequently utilized for any of a variety of machine vision tasks. In some embodiments, the 3D feature descriptor 1010 may be added to a 3D map of the local environment 1000. This 3D map may then be used by the electronic device 100 or another machine-vision enabled device for localization, whereby the electronic device 100 estimates its location, predicts what surfaces or other objects should be observable from its estimated location and its pose, captures one or more images, and then compares the 2D features extracted from the captured images with the 2D feature descriptors of the 3D feature descriptors for the objects expected to be observable to verify its estimated position or pose in a previously-visited environment. ….”, where the 3D map reads on the 3d visual representation of the environment; where the 3D map is based “a set of one or more 3D feature descriptors”; see additionally “[0099] … Moreover, for the following it is assumed that the subject object has not been previously encountered by the electronic device 100 nor has it been previously cataloged as a 3D feature descriptor in a 3D map or other 3D object database….” And “[0115] …[t]he processing system 800 uses the pose estimated at block 1302 to calculate, from a 3D map or other 3D feature database, a set of one or more 3D feature descriptors that should be visible from the estimated pose. From the 2D feature descriptors of this set of one or more 3D feature descriptors, the processing system 800 selects 2D feature descriptors having similar pose parameters as the set of observed 2D features (these selected 2D feature descriptors referred to herein as the "target 2D feature descriptors” at Moreover, Lee discloses that “different sensors may be read at different rates or frequencies. For example, an ambient light reading may be taken only once for every N image captures by the imaging cameras 114, 116, and 118, whereas a six-degrees-of-freedom (6DoF) reading from the gyroscope may be taken every image capture so as to enable detection of the relative orientation of the electronic device 100 when the corresponding image was captured. Still further, accelerometer readings may be obtained at a rate much higher than the image capture rate so as to facilitate a more accurate internal navigation determination by the electronic device” at ¶ 0052) (Lee discloses “[0070] As noted above, the electronic device 100 cycles through iterations of the method 700 to provide real-time, updated localization, mapping, and augmented reality display. However, these sub-processes do not necessarily cycle at the same rate. To illustrate, the image alignment and AR processes may update/cycle at the same frame rate as the imaging cameras 114, 116, and 118 because these processes are directly tied to the captured imagery. However, the non-image sensor capture and current context determination may proceed at different cycle rates. To illustrate, it may be appropriate to capture gyroscopic or inertial sensor states more frequently than the frame rate in order to have sufficiently accurate inertial navigation estimation. Conversely, the location-related features of the electronic device 100 may not require a high position resolution, and thus the image analysis process to determine the current position/orientation of the electronic device 100 may occur at a cycle rate slower than the frame rate of the imaging cameras.”)(emphasis added); and
matching the one or more generated feature descriptors to stored feature descriptors of the plurality of stored maps to generate a first localized pose of the electronic device (Lee discloses that “the processing system 800 compares each observed 2D feature descriptor with the closest target 2D feature descriptor (with "closest" being measured by similarity in pose parameters) to determine whether the two 2D feature descriptors match” at Fig. 13-1308, ¶ 0115).
b. Regarding claim 11, Lee discloses further comprising updating the three-dimensional visual representation by adding the tracked motion to the three-dimensional visual representation of the environment (Lee discloses that “the processing system 800 can bolster the information stored in the 3D feature descriptor by adding the observed 2D feature descriptor to the 3D feature descriptor” at Fig. 12-1214, ¶ 0112).
c. Regarding claim 12, Lee discloses wherein adding the tracked motion comprises generating linearization points based on the generated feature descriptors and solving a non-linear estimation of the three-dimensional visual representation based on the linearization points and the stored feature descriptors (Lee discloses that “the processing system 800 identifies the observed object as the target object observed from the same or similar direction. Accordingly, with the observed object so identified, at block 1214 the processing system 800 can bolster the information stored in the 3D feature descriptor by adding the observed 2D feature descriptor to the 3D feature descriptor” at Fig. 12-1212, Fig. 12-1214, ¶ 0112).
e. Regarding claim 14, Lee discloses further comprising identifying discrepancies between the matched generated feature descriptors and stored feature descriptors and minimizing the discrepancies (Lee discloses that “a "match" may be identified, for example, when the Euclidean distance between the target 2D feature descriptor and the corresponding observed 2D feature descriptor is less than a specified threshold. This match comparison may be repeated at another iteration of block 1308 for each target-observed 2D feature descriptor pairing from the corresponding sets of 2D feature descriptors” at ¶ 0115).
Regarding claims 16-18, 20, the claims are analogous and correspond to claims 10-12, 14, respectively.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 13 and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lee in view of Benihame (PGPUB 20140037140).
Regarding claim 13, Lee discloses wherein tracking motion comprises (Lee, paragraph 26, “The imaging camera 118 can be used to capture image data for the local environment 112 facing the surface 104. Further, in some embodiments, the imaging camera 118 is configured for tracking the movements of the head 122 or for facial recognition, and thus providing head tracking information that may be used to adjust a view perspective of imagery presented via the display 108.”, where movement is determined as the difference between a current position and a previous position, where both the current and previous positions are fixed)
But does not expressly disclose “estimating a current pose of the electronic device by solving a non-linear estimation of the spatial features”
Benhimane discloses “estimating a current pose of the electronic device by solving a non-linear estimation of the spatial features” (Benihamae,” [0066] Particularly, in Step 30, one or more current images are captured by a camera, the pose of which shall be determined and tracked. In Step 31, for every current image captured, features of the same types as used in the reference images are extracted. These features are called "current features". In Step 32, descriptors (or classifiers) are computed for every current feature extracted and stored. These descriptors are referenced as "current descriptors". In Step 33, the current features are matched with the reference features using the reference and current descriptors. If the descriptors are close in terms of a certain similarity measure, they are matched. For example the dot product or Euclidean distance of vector representations can be used as similarity measurement. In Step 34, given a model of the target, an outlier rejection algorithm is performed. The outlier rejection algorithm may, for example, be based on a robust pose estimation like RANSAC or PROSAC. As an output in step 34 filtered correspondences are provided which are the remaining correspondences after the outlier rejection or removal. The filtered correspondences are also provided as an input to the process according to step 40, as explained in more detail below with reference to FIG. 1b. In Step 35, the keyframe providing the highest number of verified matches is selected to be the closest keyframe. In Step 36, using the 2D coordinates of image features from the current image and the 3D coordinates indirectly known through the 2D-(2D-3D) matching, an initial camera pose P can be computed using, e.g., common linear pose estimation algorithms like DLT refined by classical non-linear optimization methods (Gauss-Newton, Levenberg-Marquard). In Step 40, a refined camera pose RP is determined according to a process as set out in FIG. 1b.”)
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to use the method of Benhimane to determine the pose of the camera in Lee.
The suggestion/motivation for doing so would have been to allow tracking when the features may be warped between the image (sensor) and 3d model.
Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results.
Therefore, it would have been obvious to combine Lee with Benhimane to obtain the invention as specified in claim 13.
Regarding claims 16-18, 20, the claims are analogous and correspond to claims 10-12, 14, respectively. See rejection of claims 10-14 for further explanation. See paragraph 22 for the processor.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to GANDHI THIRUGNANAM whose telephone number is (571)270-3261. The examiner can normally be reached M-F 8:30-5PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sumati Lefkowitz can be reached at 571-272-3638. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/GANDHI THIRUGNANAM/ Primary Examiner, Art Unit 2672 ik
Prosecution Insights