DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Applicant(s) Response to Official Action
The response filed on 12/30/2025 has been entered and made of record.
Response to Arguments/Amendments
Presented arguments have been fully considered, but some are rendered moot in view of the new ground(s) of rejection necessitated by amendment(s) initiated by the applicant(s). Examiner fully addresses below any arguments that were not rendered moot.
Claim Rejections - 35 USC § 103
Summary of Arguments:
Regarding claims 15 and 26 Applicant argues that those of ordinary skill in the industry would have had no motivation-barring improper hindsight gleaned from Applicant's own present disclosure-to have turned to the GPS tied maps allegedly utilized in Vaibhav.
Further, as amended herein, the DSM data is “predetermined data”, a construction that is not only not taught or suggested by Supun (or Vaibhav) but would have fundamentally changed the intended operation thereof.
Examiner’s Response:
Examiner respectfully disagrees. Regarding claims 15 and 26, in response to applicant’s argument that the examiner’s conclusion of obviousness is based upon improper hindsight reasoning, it must be recognized that any judgment on obviousness is in a sense necessarily a reconstruction based upon hindsight reasoning. But so long as it takes into account only knowledge which was within the level of ordinary skill at the time the claimed invention was made, and does not include knowledge gleaned only from the applicant’s disclosure, such a reconstruction is proper. See In re McLaughlin, 443 F.2d 1392, 170 USPQ 209 (CCPA 1971).
Regarding the DSM data being “predetermined data”, both Supun (Abstract) and Vaibhav (¶0024) teaches databases being access to obtain the required data, therefore said data are predetermined.
Accordingly, Examiner maintains the rejections.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 15-28 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention. Applicant has not pointed out where the new (or amended) claim is supported, nor does there appear to be a written description of the claim limitation ‘a process of matching said first image (220) and second image (230), wherein said matching process utilize said obtained DSM data (210) and said set of hypothetical camera poses’ (exemplary claim 15) in the application as filed.
When an amendment is filed in reply to an objection or rejection based on 35 U.S.C. 112(a) or pre-AIA 35 U.S.C. 112, first paragraph, a study of the entire application is often necessary to determine whether or not "new matter" is involved. Applicant should therefore specifically point out the support for any amendments made to the disclosure. MPEP 2163.06 I.
The Examiner is assuming the “matching process” has for antecedent basis a “process of matching said first image (220) and second image (230)”.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 15-21 and 23-28 are rejected under 35 U.S.C. 103 as being unpatentable over Supun Samarasekera et al. [US 20080167814 A1: already of record] in view of Vaibhav Ghadiok et al. [US 20180045519 A1: already of record].
Regarding claim 15, Supun teaches:
15. (New) A computer implemented method for determining a pose, the method (100) (i.e. A system and method for efficiently locating in 3D an object of interest in a target scene using video information captured by a plurality of cameras. The system and method provide for multi-camera visual odometry wherein pose estimates are generated for each camera by all of the cameras in the multi-camera configuration- Abstract) comprising the steps of:
capturing (110), utilizing at least one camera (311) (i.e. The system is generally illustrated in FIG. 1, which shows a number of users equipped with video cameras configured to track features (e.g., trees, vehicles, buildings) in a target scene- ¶0009… According to an embodiment of the present invention, the systems and methods are configured to integrate video-based pose estimates based on video captured by one or more video cameras- ¶0011… According to an embodiment of the invention, the system and method are configured to receive from a video camera a video snapshot comprising a pair of stereo images (i.e., a right image and a left image) related to a target scene.- ¶0019), at least a first image (220) at a first pose and a second image (230) at a second pose (i.e. . Each of the plurality of cameras is configured to capture video from that camera's perspective, herein referred to as a "local camera capture."- ¶0036);
obtaining (120) digital surface model, DSM, data (210), wherein said DSM data (210) is predetermined (i.e. the system and method can locate and identify salient landmarks in the target scene using any of the cameras in the multi-camera configuration and compare the identified landmark against a database of previously identified landmarks- Abstract… a new set of visual landmarks is extracted from the scene and the landmark database searched for their possible matches- ¶0012… The location(s) of the one or more objects of interest are recorded in a database and can be identified in a map of the environment. The video data and estimated pose information are stored in a database- ¶0016) and represents a part of Earth's surface corresponding to said first pose and said second pose (i.e. FIG. 2 illustrates an exemplary screenshot from a display communicatively connected to the vision-based navigation system of the present invention. As shown, as a user equipped with the vision-based navigation system travels through the environment, his or her position and viewpoint can be located precisely in the map, preferably in a real-time manner, as shown in the image in the upper right of FIG. 2- ¶0015);
determining (130) at least one set of hypothetical camera poses, wherein each set of hypothetical camera poses is a candidate for the first and the second poses (i.e. The captured video from each of the cameras is provided by the Camera Cluster 10 to the Visual Odometry Module 20, as shown in FIG. 4. The Visual Odometry Module 20 is a computer-based program configured to perform multi-camera visual odometry wherein pose information is transferred across different cameras of the Camera Cluster 10 such that video-based pose hypotheses generated by each camera can be evaluated by the entire system. In operation, a set of pose hypotheses (also referred to as pose candidates) are generated by each camera in the Camera Cluster 10 based on an evaluation of the local camera capture against a local dataset (i.e., the dataset of the local camera). In addition, each set of pose hypotheses is transferred to and evaluated by the other cameras in the Camera Cluster 10, resulting in the evaluation of the each camera's pose hypotheses on a global dataset and enabling the robust selection of an `optimized` pose- ¶0037 & ¶0048-0059);
calculating (140) a matching score for each set of hypothetical camera poses based a process of matching said first image (220) and second image (230) , wherein said matching process utilizes said set of hypothetical camera poses (i.e. As described above, each image is represented by a set of extracted landmarks with the HOG descriptors. Given two images taken at different locations, the Landmark Matcher 50 is configured to match the extracted landmarks between them using the HOG descriptors. According to an embodiment of the present invention, the landmark matching process comprises, for each landmark in the first image, search all the landmarks in the second image for its correspondence. The search is based on the cosine similarity score of the HOG descriptors between two landmarks. A landmark in the second image that produces the highest similarity score is considered as a potential match for the landmark in the first image. At the same time, for each landmark in the second image, a potential match in the first image can also be assigned automatically with the one that produces the highest similarity score. Finally, only the pairs that mutually have each other as the potential matches are accepted a valid match- ¶0069); and
determining (150) and estimate of the first pose based on said calculated matching scores of each set of hypothetical camera poses (i.e. The preemptive scoring in each camera is accomplished by obtaining a "global" score for each hypothesis by combining its corresponding scores determined in all the cameras (i.e., the intra-camera score and the one or more inter-camera scores) on the initial set of 100 data points from each camera. Next, a portion of the set of pose hypotheses having the lowest global scores (e.g., half of the initial set) are discarded and the remaining half is evaluated on another set of 100 points in every camera. The global scores are updated, and the process continues iteratively until an optimized pose hypothesis is identified. Upon completion of the multi-camera preemptive RANSAC process, the pose hypothesis having the highest global score is identified in each camera (herein referred to as the optimized pose hypothesis)- ¶0052).
However, Supun does not teach explicitly:
wherein matching is based on said obtained DSM data (210).
In the same field of endeavor, Vaibhav teaches:
wherein matching is based on said obtained DSM data (210) (i.e. The landmark database can include one or more maps (e.g., sparse map with landmarks for each unit region), matrix (e.g., sparse matrix), table, or other data structure. The maps can be static, be generated in real- or near-real time, or otherwise determined. Different data structures can be for different: geographic regions (e.g., overlapping, non-overlapping; covering the same or different area; etc.), landmark densities, location estimate resolution or precision, operation context (e.g., day/night), route, user account, or any other variable. Different instances of the system and/or method can use the same or different set of maps. For example, the system can reference a global map of fiducial landmarks, given a GPS estimate of vehicle pose and camera information, and generate a local map of vehicle pose given ground-truth landmarks and camera information- ¶0046).
It would have been obvious to one with ordinary skill in the art before the effective filing date of the claimed invention, to modify the teachings of Sunpun with the teachings of Vaibhav to determine a precise global system location (Vaibhav- ¶0062).
Regarding claim 16, Supun and Vaibhav teach all the limitations of claim 15 and Sunpun further teaches:
capturing (110) the first image (220) and the second image (230) comprises obtaining a relative pose difference between said first pose and said second pose; and determining (130) at least one set of hypothetical camera poses is based on said relative pose difference (i.e. In addition, conventional systems including multiple cameras or other visual sensing devices provide for limited ability to perform comprehensive visual odometry. Specifically, in such systems, visual odometry can be applied to each camera individually to estimate the pose of that camera. As such, the relative poses of the cameras (i.e., a camera's pose relative to the other cameras in the system) are fixed and known, which constrains the single-camera results. In this regard, conventional navigation systems provide for the generation of camera pose hypotheses that are based exclusively on data which is specific to the individual camera, and does not take into account the data processed by the other camera's in the multi-camera system- ¶0006… Similarly, the pose (P) of Camera k relative to Camera j can be expressed as P.sub.jk, such that: equation 2- ¶0041).
Regarding claim 17, Supun and Vaibhav teach all the limitations of claim 16 and Sunpun further teaches:
wherein obtaining the relative pose difference between the first pose and the second pose comprises determining the relative pose difference utilizing navigation information obtained from pose change tracking means (i.e. In order to increase the robustness of the Navigation System 1, the Secondary Measurement Sensors 35 (e.g., a Microelectromechanical (MEMS) Systems such as an IMU and/or a GPS unit) provides position measurement data (e.g., 3D acceleration and angular rate measurements captured by an IMU and/or a positioning reading captured by GPS) to the Integration Module 30 for integration with the video-based pose hypotheses generated by the Visual Odometry Module 20. According to an embodiment of the present invention, the Integration Module 30 can be comprised of an Extended Kalman Filter (EKF) framework. In this framework, a constant velocity and a constant angular velocity model are selected for the filter dynamics. The state vector consists of 13 elements: X, (3-vector) representing position in navigation coordinates, q, unit quaternion (4-vector) for attitude representation in navigation coordinates, v, (3-vector) for translational velocity in body coordinates, and .omega., (3-vector) for rotational velocity in body coordinates. Quaternion representation for attitude has several practical properties. Each component of the rotation matrix in quaternion is algebraic, eliminating the need for transcendental functions. It is also free of the singularities that are present with other representations and the prediction equations are treated linearly. Based on this, the process model is represented as follows- -¶0059).
Regarding claim 18, Supun and Vaibhav teach all the limitations of claim 16 and Sunpun further teaches:
wherein pose change tracking means comprise an inertial navigation system, an inertial measurement unit, and/or a system utilizing a series of images, lidar and/or radar measurements to track pose change (i.e. In order to increase the robustness of the Navigation System 1, the Secondary Measurement Sensors 35 (e.g., a Microelectromechanical (MEMS) Systems such as an IMU and/or a GPS unit) provides position measurement data (e.g., 3D acceleration and angular rate measurements captured by an IMU and/or a positioning reading captured by GPS) to the Integration Module 30 for integration with the video-based pose hypotheses generated by the Visual Odometry Module 20. According to an embodiment of the present invention, the Integration Module 30 can be comprised of an Extended Kalman Filter (EKF) framework. In this framework, a constant velocity and a constant angular velocity model are selected for the filter dynamics. The state vector consists of 13 elements: X, (3-vector) representing position in navigation coordinates, q, unit quaternion (4-vector) for attitude representation in navigation coordinates, v, (3-vector) for translational velocity in body coordinates, and .omega., (3-vector) for rotational velocity in body coordinates. Quaternion representation for attitude has several practical properties. Each component of the rotation matrix in quaternion is algebraic, eliminating the need for transcendental functions. It is also free of the singularities that are present with other representations and the prediction equations are treated linearly. Based on this, the process model is represented as follows- -¶0059).
Regarding claim 19, Supun and Vaibhav teach all the limitations of claim 15 and Sunpun further teaches:
capturing (110) the first image (220) and the second image (230) comprises obtaining pose information for said at least first pose and/or said second pose, wherein pose information comprises a pose, a position and/or a bearing; and determining (130) at least one set of hypothetical camera poses is based on said pose information (i.e. The captured video from each of the cameras is provided by the Camera Cluster 10 to the Visual Odometry Module 20, as shown in FIG. 4. The Visual Odometry Module 20 is a computer-based program configured to perform multi-camera visual odometry wherein pose information is transferred across different cameras of the Camera Cluster 10 such that video-based pose hypotheses generated by each camera can be evaluated by the entire system. In operation, a set of pose hypotheses (also referred to as pose candidates) are generated by each camera in the Camera Cluster 10 based on an evaluation of the local camera capture against a local dataset (i.e., the dataset of the local camera). In addition, each set of pose hypotheses is transferred to and evaluated by the other cameras in the Camera Cluster 10, resulting in the evaluation of the each camera's pose hypotheses on a global dataset and enabling the robust selection of an `optimized` pose- ¶0037 & ¶0048-0059).
Regarding claim 20, Supun and Vaibhav teach all the limitations of claim 19 and Sunpun further teaches:
wherein matching said first image (220) and said second image (230) is based on said pose information (i.e. In addition, conventional systems including multiple cameras or other visual sensing devices provide for limited ability to perform comprehensive visual odometry. Specifically, in such systems, visual odometry can be applied to each camera individually to estimate the pose of that camera. As such, the relative poses of the cameras (i.e., a camera's pose relative to the other cameras in the system) are fixed and known, which constrains the single-camera results. In this regard, conventional navigation systems provide for the generation of camera pose hypotheses that are based exclusively on data which is specific to the individual camera, and does not take into account the data processed by the other camera's in the multi-camera system- ¶0006… Similarly, the pose (P) of Camera k relative to Camera j can be expressed as P.sub.jk, such that: equation 2- ¶0041).
Regarding claim 21, Supun and Vaibhav teach all the limitations of claim 15 and Sunpun further teaches:
wherein determining (130) at least one set of hypothetical camera poses comprises determining at least two sets of hypothetical camera poses(i.e. The captured video from each of the cameras is provided by the Camera Cluster 10 to the Visual Odometry Module 20, as shown in FIG. 4. The Visual Odometry Module 20 is a computer-based program configured to perform multi-camera visual odometry wherein pose information is transferred across different cameras of the Camera Cluster 10 such that video-based pose hypotheses generated by each camera can be evaluated by the entire system. In operation, a set of pose hypotheses (also referred to as pose candidates) are generated by each camera in the Camera Cluster 10 based on an evaluation of the local camera capture against a local dataset (i.e., the dataset of the local camera). In addition, each set of pose hypotheses is transferred to and evaluated by the other cameras in the Camera Cluster 10, resulting in the evaluation of the each camera's pose hypotheses on a global dataset and enabling the robust selection of an `optimized` pose- ¶0037 & ¶0048-0059).
Regarding claim 23, Supun and Vaibhav teach all the limitations of claim 15 and Sunpun further teaches:
wherein determining (130) at least one set of hypothetical camera poses comprises obtaining sensor information and/or a previously determined first pose, and wherein determining (130) at least one set of hypothetical camera poses is based on said sensor information and/or said previously determined first pose (i.e. The captured video from each of the cameras is provided by the Camera Cluster 10 to the Visual Odometry Module 20, as shown in FIG. 4. The Visual Odometry Module 20 is a computer-based program configured to perform multi-camera visual odometry wherein pose information is transferred across different cameras of the Camera Cluster 10 such that video-based pose hypotheses generated by each camera can be evaluated by the entire system. In operation, a set of pose hypotheses (also referred to as pose candidates) are generated by each camera in the Camera Cluster 10 based on an evaluation of the local camera capture against a local dataset (i.e., the dataset of the local camera). In addition, each set of pose hypotheses is transferred to and evaluated by the other cameras in the Camera Cluster 10, resulting in the evaluation of the each camera's pose hypotheses on a global dataset and enabling the robust selection of an `optimized` pose- ¶0037 & ¶0048-0059);.
Regarding claim 24, Supun and Vaibhav teach all the limitations of claim 15 and Sunpun further teaches:
wherein capturing (110) at least said first image (220) and said second image (230) comprises simultaneously capturing a plurality of the at least said first image (220) and said second image (230) utilizing at least two cameras (311) (i.e. The present invention further relates to a system and method for landmark recognition. According to an embodiment of the invention, the system and method are configured to receive from a video camera a video snapshot comprising a pair of stereo images (i.e., a right image and a left image) related to a target scene- ¶0019).
Regarding claim 25, computer-readable medium storing instructions claim 25 corresponds to the same method as claimed in claim 15, and therefore is also rejected for the same reasons of obviousness as listed above.
Regarding claim 26, apparatus claim 26 is drawn to the apparatus using/performing the same method as claimed in claim 18. Therefore, apparatus claim 26 corresponds to method claim 17, and is rejected for the same reasons of obviousness as used above.
Regarding claim 27, Supun and Vaibhav teach all the limitations of claim 26 and Sunpun further teaches:
wherein the set of sensors (310) comprises at least two cameras, and wherein said set of sensors (310) is arranged to simultaneously capture at least two images with at least partial overlap (i.e. The present invention further relates to a system and method for landmark recognition. According to an embodiment of the invention, the system and method are configured to receive from a video camera a video snapshot comprising a pair of stereo images (i.e., a right image and a left image) related to a target scene- ¶0019).
Regarding claim 28, apparatus claim 28 is drawn to the apparatus using/performing the same method as claimed in claim 18. Therefore, apparatus claim 28 corresponds to method claim 18, and is rejected for the same reasons of obviousness as used above.
Allowable Subject Matter
Claim 22 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CLIFFORD HILAIRE whose telephone number is (571)272-8397. The examiner can normally be reached 5:30-1400.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, SATH V PERUNGAVOOR can be reached at (571)272-7455. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
CLIFFORD HILAIRE
Primary Examiner
Art Unit 2488
/CLIFFORD HILAIRE/Primary Examiner, Art Unit 2488