Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted was filed on 11/14/2023. The submission is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Response to Arguments
Applicant first mentions that claims 1-6 and 8 have been cancelled due to the raised 112(b) rejection and that claim 7 has been amended to overcome it. Examiner agrees and removes the 112b, however a new 112(b) has been raised for claims 13 and 19. Applicant also mentions that claim 8 also has been cancelled due to the 101 rejection, the 101 is removed. Applicant lastly argues that the newly amended claim 7, more specifically the newly added underlined section “calculate the position of the camera station and the posture of the camera based on the proportion”, examiner agrees and presents a new cited reference, Ding, which teaches the specified limitation. Therefore all of the claims remain rejected.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 13 and 19 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 13 and 19 recite the limitation "the predetermined process" . There is insufficient antecedent basis for this limitation in the claim.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 7, 9-12, 14-18 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Zamani et. al., hereafter Zamani (Zamani, Yasin, Hamed Shirzad, and Shohreh Kasaei. "Similarity measures for intersection of camera view frustums." 2017 10th Iranian Conference on Machine Vision and Image Processing (MVIP). IEEE, 2017.) in view of Ding et. al., hereafter Ding (JP Pub. No. 2019133658 A) .
As per claim 7, Zamani teaches “A computer-implemented method comprising:
determining a distance relative to a camera within a frustrum of the camera at each camera station, based at least on a position of the camera at each camera station; setting, as a target region for each camera station, a range of a predetermined shape in a projection plane at the determined distance;” (On page 3 see fig. 4 and 5, they show a region from a camera position (posture) utilizing its frustum in a projection plane. The shape is predetermined because the aspect ratio utilized is the same for the planes, also it is predetermined because it is rectangular as seen in the figures. See also section B three dimensional case pages 2-3, equations 4-8 show the region settings to calculate the distance between the position of the camera and the circumsphere center. See also page 1-2 subsection A. Two-Dimensional case along with equations 1-3 and fig. 1 and 2, which also shows setting a target region which include a range on page 2 column 1 paragraph 1 “on which the image of the scene is projected, is called image plane or near plane. The focal length or the distance from camera position to the near plane is shown with f. The measurement range of the camera is indicated by F.” The methods are predetermined since the equations are predetermined. In section 3 Similarity measures on pages 3-5, they utilize the information from section 2 to find the intersection between two camera frustums based on length, volume and probability (which also all function as region setting means). See also figs. 7, 8 and 9, which show a region intersected between each camera frustums. Zamani )
“for a given pair of images that are targeted for a distance calculation, calculating, a proportion in which the target region for the camera station from which one of the images of the given pair is taken is included in the target region for the camera station from which the other image of the image pair is taken;” (See section 3 Similarity measures on pages 3-5. All 3 subsections A, B and C, calculate a proportion based on distance between two camera regions. Subsection A shows “The relation of similarity measure based on the distance of two circumscribed circles/spheres of camera view frustum is defined as… where d is the Euclidean distance between the centers of two circles/spheres and r is the radius of them. To justify (9) see Figure 7 (this figure plotted in two-dimensional space but it is also true for three-dimensional space).” equation 9. Equation 9 shows a proportion (division). Subsection B Area/Volume shows a proportion “Since this measure should not depend on the unit of measurement, the area/volume of intersection will be divided into the area/volume of the circle/sphere (as previously mentioned, it is assumed that the internal parameters of the camera do not change over time, so the area/volume of circumscribed circles/areas are same during the time too).” The equations in this subsection both show a proportion based on distance value of the two cameras, they share the same region “the area/volume of the circumscribed circles/areas are same during the time too”. Subsection C. Probabilistic on equations 10-15, most importantly 15 also show the proportion between the two cameras (in this case similarity). Examiner also interprets “proportion” between the two regions as the similarity. See also fig. 11, shows the different proportions. See also section B. Three dimensional case on pages 2-3 which show the use of an aspect ratio, more specifically fig. 6. Zamani), however Zamani does not completely teach “and calculating a position of the camera station and a posture of the camera at the position according to the proportion.”
Ding teaches “and calculating a position of the camera station and a posture of the camera at the position according to the proportion.” (See page 4 paragraphs 2, the posture of a key frame (the other camera posture) is estimated based on a proportion between the input camera and the key frame (which is the second camera visual information) “The positioning method according to the present invention is a method for realizing position and orientation estimation in a map created in advance using an image of a predetermined location. In order to realize such positioning, the positioning method usually needs to satisfy two basic points. First, this positioning method needs to provide a complete internal representation of the environment, ie a map. The map is used for comparison with visual information obtained by positioning. Next, it is necessary to estimate the posture information of the current visual information in the map. Posture information indicates the position and orientation of a device (for example, a camera) that collects visual information. That is, the posture information should include at least position information and orientation information. Among them, the second part includes two steps. First, after determining whether or not the current visual information is already included in the map, the relative posture between the input visual information and the detected visual information is calculated. Furthermore, the absolute posture in the map is calculated based on the relative posture. Here, the relative posture of the current image means the posture of the current image with respect to other images, and the absolute position of the current image means the posture of the current image with respect to the origin set in the map. Usually, the starting point of the map is the origin.” See page 5 paragraphs 1-7 “ In step S202, feature points are extracted for each frame from successive frame images, and a feature point matching pair is obtained by performing matching on the feature points in two adjacent frame images… Meanwhile, 3D information of the extracted feature points can be acquired. The method for calculating the relative position between two frame images based on the feature point matching pair is also a known technique, and will not be described in this specification. By this step, a relative posture between two adjacent frame images is obtained. Furthermore, the relative position between any one frame image and the first frame image, that is, the absolute posture of the frame image is obtained by the power of the relative posture with the immediately preceding image.” See also page 6 last paragraph “Thereby, the positioning method shown in FIG. 1 makes it possible to obtain a relative posture of the target image with respect to each candidate key frame based on a pre-configured map. In addition, the method obtains key information of a plurality of candidate key frames in the map by performing rough matching first, and performs posture estimation based on local feature point matching.” The postures of all frames are estimated. See also page 7 “As described above, the first frame image in consecutive frame images is the origin of the map, and the i-th (i is an integer equal to or greater than 1) candidate key frame and the relative position of the first frame image, that is, the candidate key frame The absolute posture, and the relative position between the target image and the first frame image is the absolute posture of the target image. Specifically, for the absolute posture C of the i-th candidate key frame, the absolute posture of the candidate key frame is accumulated by accumulating the i relative positions of the candidate key frame and the immediately preceding frame image in the process of constructing the map.” Since each frame is shown as a different camera posture, it therefore teaches determining a posture of the other camera. See also page 2 paragraphs 1-8. Examiner also interprets “proportion” as a relationship, which presents the feature point similarity between regions of interest presented between each frame (each camera). Ding)
It would have been obvious to one of ordinary skill in the art before the effective filing
date of the claimed invention to combine the teachings of Zamani with the teachings of Ding estimate the posture of the other camera according to a proportion. The modification would have been motivated by the desire to have more accurate and effective posture estimation, therefore it is an improvement, as suggested by Ding ( See page 3 paragraph 6 “As described above, according to the positioning method according to the embodiment of the present invention, it is possible to perform matching of the region of interest using key information stored in the map constructed in advance, and to repeatedly extract feature points, and to perform matching. By performing feature point matching based on the region of interest, accurate and effective posture estimation and self-positioning can be provided even in a complicated situation. It should be noted that the above description and the detailed description to be described later are merely exemplary and are intended to further understand the present invention” See also page 7 paragraphs 1-7 “Also, according to an embodiment of the present invention, the weighting coefficient Ω is inversely proportional to the reliability of the i relative posture between the target image and the i-th candidate key frame estimated based on the feature point matching pair… the sum of feature point matching pairs by matching the target image with all i th candidate key frames. i is an integer of 1 or more. Since the relative position between the estimated target image and the i-th candidate key frame is more accurate as the reliability is higher, the weighting coefficient in the loss function is smaller”. Ding)
Claim 14 is rejected under the same analysis as claim 7. (See page 8 last 3 paragraphs. “ In another aspect of the invention, a computer readable storage medium is provided on which non-transitory computer readable instructions are stored. When a non-transitory computer-readable command is executed by the processor, the positioning method according to the embodiment of the present invention described with reference to the above drawings is executed.” Ding )
Claim 20 is rejected under the same analysis as claim 7. (See page 8 last 3 paragraphs. “ In another aspect of the invention, a computer readable storage medium is provided on which non-transitory computer readable instructions are stored. When a non-transitory computer-readable command is executed by the processor, the positioning method according to the embodiment of the present invention described with reference to the above drawings is executed.” Ding )
As per claim 9, Zamani in view of Ding teaches “The computer-implemented method of claim 7, comprising: for each camera station, setting the range of the predetermined shape in the projection plane at a predetermined distance relative to the camera.” (On page 3 see fig. 4 and 5, they show a region from a camera position (posture) utilizing its frustum in a projection plane. The shape is predetermined because the aspect ratio utilized is the same for the planes, also it is predetermined because it is rectangular as seen in the figures. See also section B three dimensional case pages 2-3, equations 4-8 show the region settings to calculate the distance between the position of the camera and the circumsphere center. The methods are predetermined since the equations are predetermined. The distance relative to the camera is predetermined since the camera parameters are predetermined as seen in section I Introduction paragraph 2 “The purpose of this paper is to introduce a similarity measure based on the geometry of the two cameras.. To do this, the intrinsic and extrinsic camera parameters need to be known. The intrinsic camera parameters are those that describe the characteristic of the camera, regardless of its position in the world such as the focal length (the distance between lens and image sensor). Moreover, the extrinsic parameters describe the location and orientation of the camera in the world space. By knowing these parameters, the camera view frustum in the global reference (for two and three-dimensional space see Figure 1 and 4 respectively) can be drawn… In other words, the view frustums of the camera at those particular times should have an intersection.” In section 3 Similarity measures on pages 3-4, they utilize the information from section 2 to find the intersection between two camera frustums based on length, volume and probability. See also figs. 7, 8 and 9, which show a region intersected between each camera frustums See also section A. Two-Dimensional Case on pages 1-2 which shows another teaching of the region setting utilizing a predetermined shape and distance. Zamani )
Claim 15 is rejected under the same analysis as claim 9.
As per claim 10, Zamani in view of Ding teaches “The computer-implemented method of claim 7, comprising: obtaining predetermined statistics of a distance from the camera at each camera station to a subject imaged by the camera at the camera station in question; and setting the range of the predetermined shape in the projection plane at a distance given by the obtained predetermined statistics relative to the camera.” (On page 3 see fig. 4 and 5, they show a region from a camera position (posture) utilizing its frustum in a projection plane. It utilizes predetermined statistics such as those present in the equations 4-8 and fig. 6. Examiner interprets “predetermined statistics” as any value or data presented. The subject imaged by the camera in this case is interpreted as the circumcenter or the intersection between cameras. The shape is predetermined because the aspect ratio utilized is the same for the planes, also it is predetermined because it is rectangular as seen in the figures. See also section B three dimensional case pages 2-3, equations 4-8 show the region settings to calculate the distance between the position of the camera and the circumsphere center. A case in which the range of the shape is given by predetermined statistics is seen in section 3 Similarity measures on page 3-5, most importantly on subsection C. Probabilistic in fig. 10 (which shows how the two centers along with distributions (also statistics) according to the center of each circumscribed circle) and fig. 11 along with equations 10-15. This subsection also shows region setting means and is within the frustum of each camera as it is also finding the similarity in the two frustums. Zamani)
Claim 16 is rejected under the same analysis as claim 10.
As per claim 11, Zamani in view of Ding teaches “The computer-implemented method of claim 7, wherein the predetermined shape is either a rectangle or an ellipse.” (See fig. 4, fig. 5, the predetermined shape includes a rectangle. Zamani)
Claim 17 is rejected under the same analysis as claim 11.
As per claim 12, Zamani in view of Ding teaches “The computer-implemented method of claim 7, wherein the predetermined shape is either a rectangle or an ellipse internally tangent to the projection plane.” (See fig. 4, fig. 5, the predetermined shape includes a rectangle internally tangent to the projection plane (inside the sphere). Zamani)
Claim 18 is rejected under the same analysis as claim 12.
Claims 13 and 19 is rejected under 35 U.S.C. 103 as being unpatentable over Zamani in view Ding and further in view of Kerl et. el. (Kerl, Christian, Jürgen Sturm, and Daniel Cremers. "Dense visual SLAM for RGB-D cameras." 2013 IEEE/RSJ international conference on intelligent robots and systems. IEEE, 2013.).
As per claim 13, Zamani in view of Ding teaches “The computer-implemented method of claim 7, wherein the predetermined process relates to a key frame in simultaneous localization and mapping.” (See page 2 paragraphs 2 and 5 “obtains candidate key information of a plurality of candidate key frames that roughly match the target image from a previously constructed map composed of key information of a plurality of key frames. Generating a region of interest of the target image, matching the region of interest with the target image based on the candidate key information, and obtaining a region of interest matching pair of at least one candidate key frame for the target image Matching the feature points in the region-of-interest matching pair based on the candidate key information to obtain a feature point matching pair of the at least one candidate key frame for the target image; based on the feature point matching pair; Calculating a relative posture between the target image and the at least one candidate key frame.” Ding), however the previously cited Kerl reference still teaches “wherein the predetermined process relates to a key frame in simultaneous localization and mapping” (See abstract and page 1 column 2 fig. 1 along with the accompanying paragraph “Fig. 1: We propose a dense SLAM method for RGB-D cameras that uses keyframes and an entropy-based loop closure detection to eliminate drift.” SLAM is simultaneous localization and mapping, and it shows the use of its key frame. Kerl)
It would have been obvious to one of ordinary skill in the art before the effective filing
date of the claimed invention to combine the teachings of Zamani and Ding with the teachings of Kerl to use a key frame related to SLAM. The modification would have been motivated by the desire to optimize intensity and depth errors as well as decrease the drift, therefore increasing accuracy, and is considered an improvement, as suggested by Kerl (See page 1 column 2 “Fig. 1: We propose a dense SLAM method for RGB-D cameras that uses keyframes and an entropy-based loop closure detection to eliminate drift… • a fast frame-to-frame registration method that optimizes both intensity and depth errors, • an entropy-based method to select keyframes, which significantly decreases the drift, • a method to validate loop closures based on the same entropy metric, and • the integration of all of the above techniques into a general graph SLAM solver that further reduces drift. Through extensive evaluation on a publicly available RGB-D benchmark [10], we demonstrate that our approach achieves higher accuracy on average than existing feature-based methods” Kerl)
Claim 19 is rejected under the same analysis as claim 13.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DYLAN J MENDEZ MUNIZ whose telephone number is (703)756-5672. The examiner can normally be reached M-F, 8AM - 5PM ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Moyer can be reached at (571) 272-9523. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DYLAN JOHN MENDEZ MUNIZ/Examiner, Art Unit 2675
/ANDREW M MOYER/Supervisory Patent Examiner, Art Unit 2675
Prosecution Insights