Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statements (IDS) submitted were filed on 11/14/2023 and 06/08/2026. The submission is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 09/10/2026 has been entered.
Response to Arguments
Applicant has filed an RCE (Request for Continued Examination) and proposes amendments to the claims, the first issue mentioned is that claims 13 and 19 were amended in order to fix the 112(b) rejection, examiner agrees and withdraws the rejection. Applicant then mentions that independent claims 7, 14 and 20 have been amended to recite “by selecting a key frame based on the calculated distance value and estimating the posture of the camera using the selected key frame.”, with the argument being that cited prior art Zamani and Ding do not teach this newly added amendment, examiner agrees and provides new cited art Leung that when combined with the previous prior art does teach the whole independent claims along with geometric characteristics utilized for proportions. Examiner also makes note of claims 19 and 13, which were previously rejected by cited art Kerl and after further consideration, are now rejected under the previously cited art Ding. All of the claims stay rejected.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 7, 9-20 are rejected under 35 U.S.C. 103 as being unpatentable over Zamani et. al., hereafter Zamani (Zamani, Yasin, Hamed Shirzad, and Shohreh Kasaei. "Similarity measures for intersection of camera view frustums." 2017 10th Iranian Conference on Machine Vision and Image Processing (MVIP). IEEE, 2017.) in view of Ding et. al., hereafter Ding (JP Pub. No. 2019133658 A) and further in view of Leung, Ka Ming (US Pub. No. 20150098645 A1) .
As per claim 7, Zamani teaches “A computer-implemented method comprising:
determining a distance relative to a camera within a frustrum of the camera at each camera station, based at least on a position of the camera at each camera station; setting, as a target region for each camera station, a range of a predetermined shape in a projection plane at the determined distance;” (On page 3 see fig. 4 and 5, they show a region from a camera position (posture) utilizing its frustum in a projection plane. The shape is predetermined because the aspect ratio utilized is the same for the planes, also it is predetermined because it is rectangular as seen in the figures. See also section B three dimensional case pages 2-3, equations 4-8 show the region settings to calculate the distance between the position of the camera and the circumsphere center. See also page 1-2 subsection A. Two-Dimensional case along with equations 1-3 and fig. 1 and 2, which also shows setting a target region which include a range on page 2 column 1 paragraph 1 “on which the image of the scene is projected, is called image plane or near plane. The focal length or the distance from camera position to the near plane is shown with f. The measurement range of the camera is indicated by F.” The methods are predetermined since the equations are predetermined. In section 3 Similarity measures on pages 3-5, they utilize the information from section 2 to find the intersection between two camera frustums based on length, volume and probability (which also all function as region setting means). See also figs. 7, 8 and 9, which show a region intersected between each camera frustums. Zamani )
“for a given pair of images that are targeted for a distance calculation, calculating, as a distance value, a proportion in which the target region for the camera station from which one of the images of the given pair is taken is included in the target region for the camera station from which the other image of the image pair is taken;” (See section 3 Similarity measures on pages 3-5. All 3 subsections A, B and C, calculate a proportion based on distance between two camera regions. Examiner interprets “distance value” as any result and process containing distance. Subsection A shows “The relation of similarity measure based on the distance of two circumscribed circles/spheres of camera view frustum is defined as… where d is the Euclidean distance between the centers of two circles/spheres and r is the radius of them. To justify (9) see Figure 7 (this figure plotted in two-dimensional space but it is also true for three-dimensional space).” equation 9. Equation 9 shows a proportion (division). Subsection B Area/Volume shows a proportion “Since this measure should not depend on the unit of measurement, the area/volume of intersection will be divided into the area/volume of the circle/sphere (as previously mentioned, it is assumed that the internal parameters of the camera do not change over time, so the area/volume of circumscribed circles/areas are same during the time too).” The equations in this subsection both show a proportion based on distance value of the two cameras, they share the same region “the area/volume of the circumscribed circles/areas are same during the time too”. Subsection C. Probabilistic on equations 10-15, most importantly 15 also show the proportion between the two cameras (in this case similarity). Examiner also interprets “proportion” between the two regions as the similarity. See also fig. 11, shows the different proportions. See also section B. Three dimensional case on pages 2-3 which show the use of an aspect ratio, more specifically fig. 6. Zamani), however Zamani does not completely teach “and calculating a position of the camera station and a posture of the camera at the position according to the proportion by selecting a key frame based on the calculated distance value and estimating the posture of the camera using the selected key frame.”
Ding teaches “and calculating a position of the camera station and a posture of the camera at the position according to the proportion… and estimating the posture of the camera using the… key frame” (See page 4 paragraphs 2, the posture of a key frame (the other camera posture) is estimated based on a proportion between the input camera and the key frame (which is the second camera visual information) “The positioning method according to the present invention is a method for realizing position and orientation estimation in a map created in advance using an image of a predetermined location. In order to realize such positioning, the positioning method usually needs to satisfy two basic points. First, this positioning method needs to provide a complete internal representation of the environment, ie a map. The map is used for comparison with visual information obtained by positioning. Next, it is necessary to estimate the posture information of the current visual information in the map. Posture information indicates the position and orientation of a device (for example, a camera) that collects visual information. That is, the posture information should include at least position information and orientation information. Among them, the second part includes two steps. First, after determining whether or not the current visual information is already included in the map, the relative posture between the input visual information and the detected visual information is calculated. Furthermore, the absolute posture in the map is calculated based on the relative posture. Here, the relative posture of the current image means the posture of the current image with respect to other images, and the absolute position of the current image means the posture of the current image with respect to the origin set in the map. Usually, the starting point of the map is the origin.” See page 5 paragraphs 1-7 “ In step S202, feature points are extracted for each frame from successive frame images, and a feature point matching pair is obtained by performing matching on the feature points in two adjacent frame images… Meanwhile, 3D information of the extracted feature points can be acquired. The method for calculating the relative position between two frame images based on the feature point matching pair is also a known technique, and will not be described in this specification. By this step, a relative posture between two adjacent frame images is obtained. Furthermore, the relative position between any one frame image and the first frame image, that is, the absolute posture of the frame image is obtained by the power of the relative posture with the immediately preceding image.” See also page 6 last paragraph “Thereby, the positioning method shown in FIG. 1 makes it possible to obtain a relative posture of the target image with respect to each candidate key frame based on a pre-configured map. In addition, the method obtains key information of a plurality of candidate key frames in the map by performing rough matching first, and performs posture estimation based on local feature point matching.” The postures of all frames are estimated. See also page 7 “As described above, the first frame image in consecutive frame images is the origin of the map, and the i-th (i is an integer equal to or greater than 1) candidate key frame and the relative position of the first frame image, that is, the candidate key frame The absolute posture, and the relative position between the target image and the first frame image is the absolute posture of the target image. Specifically, for the absolute posture C of the i-th candidate key frame, the absolute posture of the candidate key frame is accumulated by accumulating the i relative positions of the candidate key frame and the immediately preceding frame image in the process of constructing the map.” Since each frame is shown as a different camera posture, it therefore teaches determining a posture of the other camera. See also page 2 paragraphs 1-8. Examiner also interprets “proportion” as a relationship, which presents the feature point similarity between regions of interest presented between each frame (each camera). Ding)
It would have been obvious to one of ordinary skill in the art before the effective filing
date of the claimed invention to combine the teachings of Zamani with the teachings of Ding to estimate the posture of the other camera according to a proportion. The modification would have been motivated by the desire to have more accurate and effective posture estimation, therefore it is an improvement, as suggested by Ding ( See page 3 paragraph 6 “As described above, according to the positioning method according to the embodiment of the present invention, it is possible to perform matching of the region of interest using key information stored in the map constructed in advance, and to repeatedly extract feature points, and to perform matching. By performing feature point matching based on the region of interest, accurate and effective posture estimation and self-positioning can be provided even in a complicated situation. It should be noted that the above description and the detailed description to be described later are merely exemplary and are intended to further understand the present invention” See also page 7 paragraphs 1-7 “Also, according to an embodiment of the present invention, the weighting coefficient Ω is inversely proportional to the reliability of the i relative posture between the target image and the i-th candidate key frame estimated based on the feature point matching pair… the sum of feature point matching pairs by matching the target image with all i th candidate key frames. i is an integer of 1 or more. Since the relative position between the estimated target image and the i-th candidate key frame is more accurate as the reliability is higher, the weighting coefficient in the loss function is smaller”. Ding)
Leung teaches “…according to the proportion by selecting a key frame based on the calculated distance value and estimating the posture of the camera using the selected key frame.” (See paragraph 90 “[0090] In one arrangement, a five-point algorithm is used at step 350 to determine the relative pose of the camera 120 for the first and second keyframes. The five-point algorithm may be used to randomly select five image feature correspondences across the first and the second keyframes to determine an estimate of an essential matrix. Each time an estimate of the essential matrix is generated, an objective function, such as the Sampson distance measure (SD), is used to evaluate the quality of the estimation (i.e. residual errors) using at least a thousand or more image feature correspondences. The five-point algorithm may be repeatedly executed a large number of times for different random sets of image feature correspondences. The essential matrix that gives a lowest score (i.e. min SD) may be selected as a best estimate for the relative pose of the camera 120 for the first and second keyframes…” See paragraphs 130-140 “[0130] Following step 620, the method 600 proceeds to selecting step 630. At step 630, the mapper software module 1024, under execution of the processor 205, selects an existing keyframe to pair with the new keyframe selected at step 620 for triangulation. For example, FIG. 9 shows an existing keyframe 901 and a new keyframe 902 representing two views of a scene 110. The existing keyframe 901 is represented by the sum of the regions A and C (i.e., A+C). The new keyframe 902 consists of an unmapped region 903 determined at step 550 by the tracker software module 1022… [0131] The method 600 is configured for then determining the ratio between the portion of the projected unmapped region that falls within the existing keyframe (i.e., A) and the entire area of the projected unmapped region (i.e., A+B) is computed (i.e., A/A+B). The keyframe that has the largest overlap corresponding to the maximum connected area of unmapped region identified at step 550 is then selected. Accordingly, the keyframe is selected based on at least one of the size and the ratio… [0132]… For each remaining salient image feature, the matching image feature in the selected keyframe is determined by performing an epipolar search. Determined image features in the selected keyframe a short distance away from an epipolar line are compared with the salient image feature using ZSSD. If a match is found, the new point is triangulated and inserted into the map 190. If a match is found, the new point is triangulated and inserted into the map 190. The comparison of the image features in the selected keyframe repeats until all matching correspondences are added to the 3D point cloud 192 and the method 600 returns to step 610. ” See also paragraphs 141-150 “[0143] (i) Score.sub.bd is a measure of the baseline distance between the locations of an existing keyframe 901 and the new keyframe 902. As seen in FIG. 8A, the value of the Score.sub.bd peaks at one-tenth of the estimated scene depth (i.e., depth of the scene 110) which may be determined from the distances of all found map points to the camera location of the new keyframe 902…” “[0147] As shown in the example of FIG. 9, the projected unmapped region 907 is only partially overlapped with the existing keyframe 901, denoted by "A". In FIG. 9, "B" represents the portion of the unmapped region that is outside the existing keyframe 901. In one arrangement, Score.sub.ork is related to the percentage of the existing keyframe 901 of the unmapped region A. Score.sub.ork increases from zero (0) to one (1) as the percentage of the existing keyframe 901 of the unmapped region A increases from 0 to 40%. Score.sub.ork then reduces down to zero (0) as the percentage of the existing keyframe 901 of the unmapped region A reaches 50%, because a percentage of 50% implies that the pose estimated for the existing keyframe 901 might be inaccurate.” “0148] (v) Score.sub.orf is related to the ratio between the portion of the projected unmapped region that falls within the existing keyframe (i.e., A) and the entire area of the projected unmapped region (i.e., A+B)… [0150] Once the scores are determined for all existing keyframes, the method 700 then proceeds to step 750, where the mapper software module 1024, under execution of the processor 205, selects the existing keyframe with the highest score to pair with the new keyframe.” By selecting the highest score, it is making sure that the selected keyframe contains most accurate pose. See also paragraphs 7-9 and 110-115. See also figs. 7, 8 and 9. Leung)
It would have been obvious to one of ordinary skill in the art before the effective filing
date of the claimed invention to combine the teachings of Zamani with the teachings of Ding and Leung to estimate the posture of the other camera according to a proportion by selecting a key frame along with estimating the posture with the selected key frame. The modification would have been motivated by the desire to have more accurate and effective posture estimation, ensure valid triangulation, along with fast and efficient map extension and better quality, therefore it is an improvement, as suggested by Leung ( See paragraph 9 “[0009] To ensure valid triangulation and maximize the number of new map points, one known method firstly determines the closest point of intersection of the camera viewing vectors. The distance between the point of intersection and the camera locations is then compared to scene depths for the keyframes. The difference between expected point depth and actual depth is used as a quality measure…” See paragraph 2 “0002] The present invention relates generally to augmented reality (AR) systems and, in particular, to fast and efficient map expansion. The present invention also relates to a method, apparatus and system for selecting a frame from a plurality of video image frames…”. See paragraph 147 and 140-150 “[0147] As shown in the example of FIG. 9, the projected unmapped region 907 is only partially overlapped with the existing keyframe 901, denoted by "A". In FIG. 9, "B" represents the portion of the unmapped region that is outside the existing keyframe 901. In one arrangement, Score.sub.ork is related to the percentage of the existing keyframe 901 of the unmapped region A. Score.sub.ork increases from zero (0) to one (1) as the percentage of the existing keyframe 901 of the unmapped region A increases from 0 to 40%. Score.sub.ork then reduces down to zero (0) as the percentage of the existing keyframe 901 of the unmapped region A reaches 50%, because a percentage of 50% implies that the pose estimated for the existing keyframe 901 might be inaccurate. The pose estimated for the existing keyframe 901 might be inaccurate for a percentage of 50% as the found map points are localised in a relatively small image region of the existing keyframe 901.” By selecting the highest score, it is making sure that the selected keyframe contains most accurate pose. Leung)
Claim 14 is rejected under the same analysis as claim 7. (See page 8 last 3 paragraphs. “ In another aspect of the invention, a computer readable storage medium is provided on which non-transitory computer readable instructions are stored. When a non-transitory computer-readable command is executed by the processor, the positioning method according to the embodiment of the present invention described with reference to the above drawings is executed.” Ding. See also paragraph 50. Leung)
Claim 20 is rejected under the same analysis as claim 7. (See page 8 last 3 paragraphs. “ In another aspect of the invention, a computer readable storage medium is provided on which non-transitory computer readable instructions are stored. When a non-transitory computer-readable command is executed by the processor, the positioning method according to the embodiment of the present invention described with reference to the above drawings is executed.” Ding. See also paragraph 50. Leung.)
As per claim 9, Zamani in view of Ding and Leung teaches “The computer-implemented method of claim 7, comprising: for each camera station, setting the range of the predetermined shape in the projection plane at a predetermined distance relative to the camera.” (On page 3 see fig. 4 and 5, they show a region from a camera position (posture) utilizing its frustum in a projection plane. The shape is predetermined because the aspect ratio utilized is the same for the planes, also it is predetermined because it is rectangular as seen in the figures. See also section B three dimensional case pages 2-3, equations 4-8 show the region settings to calculate the distance between the position of the camera and the circumsphere center. The methods are predetermined since the equations are predetermined. The distance relative to the camera is predetermined since the camera parameters are predetermined as seen in section I Introduction paragraph 2 “The purpose of this paper is to introduce a similarity measure based on the geometry of the two cameras.. To do this, the intrinsic and extrinsic camera parameters need to be known. The intrinsic camera parameters are those that describe the characteristic of the camera, regardless of its position in the world such as the focal length (the distance between lens and image sensor). Moreover, the extrinsic parameters describe the location and orientation of the camera in the world space. By knowing these parameters, the camera view frustum in the global reference (for two and three-dimensional space see Figure 1 and 4 respectively) can be drawn… In other words, the view frustums of the camera at those particular times should have an intersection.” In section 3 Similarity measures on pages 3-4, they utilize the information from section 2 to find the intersection between two camera frustums based on length, volume and probability. See also figs. 7, 8 and 9, which show a region intersected between each camera frustums See also section A. Two-Dimensional Case on pages 1-2 which shows another teaching of the region setting utilizing a predetermined shape and distance. Zamani )
Claim 15 is rejected under the same analysis as claim 9.
As per claim 10, Zamani in view of Ding and Leung teaches “The computer-implemented method of claim 7, comprising: obtaining predetermined statistics of a distance from the camera at each camera station to a subject imaged by the camera at the camera station in question; and setting the range of the predetermined shape in the projection plane at a distance given by the obtained predetermined statistics relative to the camera.” (On page 3 see fig. 4 and 5, they show a region from a camera position (posture) utilizing its frustum in a projection plane. It utilizes predetermined statistics such as those present in the equations 4-8 and fig. 6. Examiner interprets “predetermined statistics” as any value or data presented. The subject imaged by the camera in this case is interpreted as the circumcenter or the intersection between cameras. The shape is predetermined because the aspect ratio utilized is the same for the planes, also it is predetermined because it is rectangular as seen in the figures. See also section B three dimensional case pages 2-3, equations 4-8 show the region settings to calculate the distance between the position of the camera and the circumsphere center. A case in which the range of the shape is given by predetermined statistics is seen in section 3 Similarity measures on page 3-5, most importantly on subsection C. Probabilistic in fig. 10 (which shows how the two centers along with distributions (also statistics) according to the center of each circumscribed circle) and fig. 11 along with equations 10-15. This subsection also shows region setting means and is within the frustum of each camera as it is also finding the similarity in the two frustums. Zamani)
Claim 16 is rejected under the same analysis as claim 10.
As per claim 11, Zamani in view of Ding and Leung teaches “The computer-implemented method of claim 7, wherein the predetermined shape is either a rectangle or an ellipse.” (See fig. 4, fig. 5, the predetermined shape includes a rectangle. Zamani)
Claim 17 is rejected under the same analysis as claim 11.
As per claim 12, Zamani in view of Ding and Leung teaches “The computer-implemented method of claim 7, wherein the predetermined shape is either a rectangle or an ellipse internally tangent to the projection plane.” (See fig. 4, fig. 5, the predetermined shape includes a rectangle internally tangent to the projection plane (inside the sphere). Zamani)
Claim 18 is rejected under the same analysis as claim 12.
As per claim 13, Zamani in view of Ding and Leung already teaches “The computer-implemented method of claim 7, wherein calculating the position of the camera station and the posture of the camera relate to a key frame…” however Ding also teaches “in simultaneous localization and mapping.” (See page 2 paragraphs 2-5 ”SLAM is a subject that is currently attractin location estimation. SLAM integrates the positioning of a moving target and the creation of the environment map…the present invention is a method for realizing self-positioning based on an existing map and visual information…” “obtains candidate key information of a plurality of candidate key frames that roughly match the target image from a previously constructed map composed of key information of a plurality of key frames. Generating a region of interest of the target image, matching the region of interest with the target image based on the candidate key information, and obtaining a region of interest matching pair of at least one candidate key frame for the target image Matching the feature points in the region-of-interest matching pair based on the candidate key information to obtain a feature point matching pair of the at least one candidate key frame for the target image; based on the feature point matching pair; Calculating a relative posture between the target image and the at least one candidate key frame.” Ding), In addition, the details related in reference Leung in the rejection of claim 1 are also interpreted as simultaneous localization and mapping.
Claim 19 is rejected under the same analysis as claim 13.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DYLAN J MENDEZ MUNIZ whose telephone number is (703)756-5672. The examiner can normally be reached M-F, 8AM - 5PM ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Vu Le can be reached at (571) 272-7332. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DYLAN JOHN MENDEZ MUNIZ/Examiner, Art Unit 2675
/VU LE/Supervisory Patent Examiner, Art Unit 2668
Prosecution Insights