CTNF 18/749,436 CTNF 87565 DETAILED ACTION Notice of Pre-AIA or AIA Status 07-03-aia AIA 15-10-aia The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA. Claim Objections 07-29-01 AIA Claim s 2 and 5-6 are objected to because of the following informalities: Claim 2, line 3, appears to be missing an “and” or an “or” at the end of the limitation. Claim 5, line 3, appears to be missing an “and” or an “or” at the end of the limitation. Claim 6, line 1, recites “the method of any one of claim 3” which appears to be incomplete (e.g., should recite “the method of any one of claims 3-5”) . Appropriate correction is required. Claim Rejections - 35 USC § 102 07-06 AIA 15-10-15 In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status. 07-07-aia AIA 07-07 The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action: A person shall be entitled to a patent unless – 07-08-aia AIA (a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention. 07-15-aia AIA Claim(s) 1-5, 11, and 18-19 is/are rejected under 35 U.S.C. 102 (a)(1) as being anticipated by Fujiki et al., US 2013/0121592 . Regarding claim 1 , Fujiki discloses a method for determining a spatial position of an object using an image calibrated relative to a reference frame (para 0001 and 0024; a position and orientation measurement apparatus, a position and orientation measurement method, and a storage medium; a position and orientation measurement apparatus that estimates the position and orientation of an object by fitting a known three-dimensionally shaped model of an object to the image obtained by capturing the image of the object ) , the method comprising: obtaining a candidate region associated with the object in the reference frame (para 0037-0038; In step S301, initial setting of data used in the processing is performed. Specifically, three-dimensional model data of the measurement target object, approximate value data of the position and orientation of the measurement target object; A three-dimensional model is defined by a set of vertices and a set of line segments connecting vertices. Accordingly, the three-dimensional model data is configured by the identification numbers and the coordinates of the vertices, the identification numbers of the line segments, and the identification numbers of vertices on the ends of the line segments ) ; projecting the candidate region into the calibrated image (para 0010 and 0046; projection means for projecting a line segment that constitutes the three- dimensional model onto the two-dimensional image based on approximate values of position and orientation of the target object ) ; adjusting the projected candidate region based on an image element associated with the object in the calibrated image (fig. 3; para 0042-0044; In step S304, the association unit 253 associates the model of the measurement target object with an image feature; In step S305, the optimization processing unit 254 corrects the approximate position and orientation of the measurement target object, thereby calculating the position and orientation of the measurement target object; line segments that constitute the three-dimensional model are associated with edges on the two-dimensional image ) ; and determining a spatial position of the object using the adjusted projected candidate region (para 0043, 0059, and 0068; the optimization processing unit 254 corrects the approximate position and orientation of the measurement target object, thereby calculating the position and orientation of the measurement target object ) . Regarding claim 2 , the method of claim 1, Fujiki further discloses wherein the image element is at least one of: a bounding-box enclosing the object in the calibrated image; a set of pixels in the calibrated image identified as belonging to the object; a set of edges in the calibrated image identified as representing the object (para 0025 and 0044) . Regarding claim 3 , the method of claim 1, Fujiki further discloses wherein obtaining a candidate region associated with the object in the reference frame comprises obtaining one candidate region associated with the object in the reference frame, and wherein the one candidate region is determined based on an approximate position and on a candidate orientation of the object (para 0037 and 0039) . Regarding claim 4 , the method of claim 3, Fujiki further discloses wherein the obtaining step, the projecting step, the adjusting step and the determining step are reiterated until the fulfilment of a completion condition, the determined spatial position for a current iteration being used as the approximate position for the new iteration (para 0039, 0059, and 0069) . Regarding claim 5 , the method of claim 4, Fujiki further discloses wherein the completion condition is fulfilled if one or more of the following conditions is/are met: a predefined number of iterations is reached; the distance between two successively determined spatial positions is less than a predefined distance (para 0039, 0059, and 0069) . Regarding claim 11 , the method of claim 1, Fujiki further discloses wherein determining a candidate region associated with the object in the reference frame comprises determining a candidate region associated with the object in the reference frame using the calibrated image (para 0039) . Regarding claim 18 , this claim recites substantially the same limitations that are performed by claim 1 above, and it is rejected for the same reasons. Regarding claim 19 , this claim recites substantially the same limitations that are performed by claim 1 above, and it is rejected for the same reasons . Claim Rejections - 35 USC § 103 07-06 AIA 15-10-15 In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status. 07-20-aia AIA The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action: A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made. 07-23-aia AIA The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows: 1. Determining the scope and contents of the prior art. 2. Ascertaining the differences between the prior art and the claims at issue. 3. Resolving the level of ordinary skill in the pertinent art. 4. Considering objective evidence present in the application indicating obviousness or nonobviousness. 07-21-aia AIA Claim (s) 6 and 16-17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Fujiki et al., US 2013/0121592 in view of Grabner et al., US 2019/0147221 . Regarding claim 6 , the method of any one of claim 3, Fujiki does not explicitly disclose wherein the candidate orientation of the object is determined using two successively captured calibrated images as claimed. However, Grabner discloses the process 1200 can receive user input requesting movement of the object (represented by the three-dimensional model) from a first location (and/or from a first pose) to a second location (and/or to a second pose). The process 1200 can include obtaining an additional input image. The additional input image includes the object in a different pose, in a different location, or both a different pose and location than a pose and/or location of the object in the input image that was obtained at block 1202). The process 1200 can adjust one or more of the pose or the location of the candidate three-dimensional model in an output image based on a difference between the pose or location of the object in the additional input image and the pose or location of the object in the input image (para 0133-0134) . Therefore, taking the combined disclosures of Fujiki and Grabner as a whole, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the process 1200 can receive user input requesting movement of the object (represented by the three-dimensional model) from a first location (and/or from a first pose) to a second location (and/or to a second pose). The process 1200 can include obtaining an additional input image. The additional input image includes the object in a different pose, in a different location, or both a different pose and location than a pose and/or location of the object in the input image that was obtained at block 1202). The process 1200 can adjust one or more of the pose or the location of the candidate three-dimensional model in an output image based on a difference between the pose or location of the object in the additional input image and the pose or location of the object in the input image as taught by Grabner into the invention of Fujiki for the benefit of adjusting the location of the three-dimensional model from the first location (and/or from the first pose) to the second location (and/or to the second pose) in an output image (Grabner: para 0133) . Regarding claim 16 , the method of claim 1, Fujiki discloses wherein determining a spatial position of the object using an adjusted projected candidate region (para 0043, 0059, and 0068) . Fujiki discloses claim 16 as enumerated above, but Fujiki does not explicitly disclose determining a spatial position of the object in the calibrated image using the adjusted projected candidate region; and re-projecting the determined spatial position to the reference frame as claimed. However, Grabner discloses the 2D points in this example are the 2D projections estimated from an input image using the regressor CNN. The PnP algorithm estimates a pose of a calibrated camera (relative to a target object in a certain pose) using a given set of n 3D points of the object's bounding box in world coordinates and their corresponding 2D projections in the image. In some examples, the refiner method includes training a regressor (e.g., a convolutional neural network) to update the 2D projections of the bounding box of an object. The regressor used to update the 2D projections may include a different regressor than the regressor 404. The 2D projections can be updated by comparing an input image to a rendering of the object for an initial pose estimate (para 0153-0160) . Therefore, taking the combined disclosures of Fujiki and Grabner as a whole, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the 2D points in this example are the 2D projections estimated from an input image using the regressor CNN. The PnP algorithm estimates a pose of a calibrated camera (relative to a target object in a certain pose) using a given set of n 3D points of the object's bounding box in world coordinates and their corresponding 2D projections in the image. In some examples, the refiner method includes training a regressor (e.g., a convolutional neural network) to update the 2D projections of the bounding box of an object. The regressor used to update the 2D projections may include a different regressor than the regressor 404. The 2D projections can be updated by comparing an input image to a rendering of the object for an initial pose estimate as taught by Grabner into the invention of Fujiki for the benefit of updating to improve the pose (Grabner: para 0160) . Regarding claim 17 , the method of claim 1, Fujiki discloses wherein determining a spatial position of the object using an adjusted projected candidate region (para 0043, 0059, and 0068) . Fujiki discloses claim 17 as enumerated above, but Fujiki does not explicitly disclose re-projecting the adjusted projected candidate region to the reference frame; and determining the spatial position of the object using the re-projected adjusted candidate region as claimed. However, Grabner discloses the 2D points in this example are the 2D projections estimated from an input image using the regressor CNN. The PnP algorithm estimates a pose of a calibrated camera (relative to a target object in a certain pose) using a given set of n 3D points of the object's bounding box in world coordinates and their corresponding 2D projections in the image. In some examples, the refiner method includes training a regressor (e.g., a convolutional neural network) to update the 2D projections of the bounding box of an object. The regressor used to update the 2D projections may include a different regressor than the regressor 404. The 2D projections can be updated by comparing an input image to a rendering of the object for an initial pose estimate (para 0153-0160) . Therefore, taking the combined disclosures of Fujiki and Grabner as a whole, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the 2D points in this example are the 2D projections estimated from an input image using the regressor CNN. The PnP algorithm estimates a pose of a calibrated camera (relative to a target object in a certain pose) using a given set of n 3D points of the object's bounding box in world coordinates and their corresponding 2D projections in the image. In some examples, the refiner method includes training a regressor (e.g., a convolutional neural network) to update the 2D projections of the bounding box of an object. The regressor used to update the 2D projections may include a different regressor than the regressor 404. The 2D projections can be updated by comparing an input image to a rendering of the object for an initial pose estimate as taught by Grabner into the invention of Fujiki for the benefit of updating to improve the pose (Grabner: para 0160) . 07-21-aia AIA Claim (s) 7-9 and 12-15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Fujiki et al., US 2013/0121592 in view of Lee et al., US 11,373,332 . Regarding claim 7 , the method of claim 1, Fujiki does not explicitly disclose wherein obtaining a candidate region associated with the object in the reference frame comprises obtaining a plurality of candidate regions associated with the object in the reference frame, the plurality of candidate regions being determined based on an approximate position of the object and each of them being oriented along a respective candidate orientation, and wherein the projecting, adjusting and determining steps are carried out for each candidate region of the plurality of candidate regions, the method further comprising a step including determining a final spatial position of the object based on the determined spatial positions as claimed. However, Lee discloses when a pose is estimated from data that contains many outliers, a Random sample consensus (RANSAC) technique can be performed for robust model fitting. The pose determination engine 910 can determine the best 6D pose hypothesis from among a plurality of 6D pose hypotheses using RANSAC or other suitable technique. The final object localization result for a detected object includes best 6D pose hypothesis selected using RANSAC or other technique (col. 8, lines 55-58 and col. 17, line 65-col. 18, line 3) . Therefore, taking the combined disclosures of Fujiki and Lee as a whole, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate when a pose is estimated from data that contains many outliers, a Random sample consensus (RANSAC) technique can be performed for robust model fitting. The pose determination engine 910 can determine the best 6D pose hypothesis from among a plurality of 6D pose hypotheses using RANSAC or other suitable technique. The final object localization result for a detected object includes best 6D pose hypothesis selected using RANSAC or other technique as taught by Lee into the invention of Fujiki for the benefit of performing point-based object localization to determine the pose of an object in an image (Lee: col. 1, lines 15-18) . Regarding claim 8 , the method of claim 7, Fujiki in the combination further disclose wherein the final spatial position of the object is determined by applying a uniform averaging to the determined spatial positions (para 0004, 0032, and 0079) . Regarding claim 9 , the method of claim 7, Fujiki in the combination further disclose wherein a weight is assigned to each of the plurality of candidate orientations, and wherein the final spatial position of the object is determined by applying a weighted averaging to the determined spatial positions (para 0004, 0025, 0032, and 0079) . Regarding claim 12 , the method of claim 1, Fujiki does not explicitly disclose wherein determining a candidate region associated with the object in the reference frame comprises determining a candidate region associated with the object in the reference frame using one or more geometric characteristics of the ground on which the object is located as claimed. However, Lee discloses in addition to the 3D model 1007 of the object and its 2D-3D sample point (or keypoint) correspondences, which are provided as input by default in the PnP problem, the 2D bounding box of the detected object is provided as input to the vector determination engine 906. The vector determination engine 906 can project the rays of both sides of the 2D bounding box on a horizontal plane (e.g., a road surface). The sample point can be redefined as an origin in the object coordinate system of the 3D model 1007 (col. 18, lines 6-33) . Therefore, taking the combined disclosures of Fujiki and Lee as a whole, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate in addition to the 3D model 1007 of the object and its 2D-3D sample point (or keypoint) correspondences, which are provided as input by default in the PnP problem, the 2D bounding box of the detected object is provided as input to the vector determination engine 906. The vector determination engine 906 can project the rays of both sides of the 2D bounding box on a horizontal plane (e.g., a road surface). The sample point can be redefined as an origin in the object coordinate system of the 3D model 1007 as taught by Lee into the invention of Fujiki for the benefit of performing point-based object localization to determine the pose of an object in an image (Lee: col. 1, lines 15-18) . Regarding claim 13 , the method of claim 3, Fujiki does not explicitly disclose wherein each candidate region is centered at the approximate position of the object as claimed. However, Lee discloses for example, it can be assumed that the 3D model 1007 (and the 2D-3D sample point or keypoint correspondences of the 3D model) and a pitch angle (the angle from the camera center to the ground plane or horizontal plane) are already known as prior information (col. 18, lines 6-33) . Therefore, taking the combined disclosures of Fujiki and Lee as a whole, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate for example, it can be assumed that the 3D model 1007 (and the 2D-3D sample point or keypoint correspondences of the 3D model) and a pitch angle (the angle from the camera center to the ground plane or horizontal plane) are already known as prior information as taught by Lee into the invention of Fujiki for the benefit of performing point-based object localization to determine the pose of an object in an image (Lee: col. 1, lines 15-18) . Regarding claim 14 , the method of claim 2, Fujiki does not explicitly disclose wherein the image element is the bounding-box enclosing the reference object in the calibrated image, and wherein adjusting a projected candidate region comprises adjusting the projected candidate region in a way to increase the overlapping between the projected candidate region and the bounding-box as claimed. However, Lee discloses using the prior information, the problem can be redefined as aligning the projection of a 3D bounding box of the object into the back-projected rays of both sides of the 2D bounding box in a bird-eye view (BEV) of the object (i.e., from directly above the object. A unique pose hypotheses per sample point can be obtained by solving the equation. The RANSAC process (e.g., the algorithm shown in FIG. 7) or other suitable technique can be used to determine the optimal pose parameter with the maximum number of inliers among those hypotheses. In some cases, the determined pose can then be optimized (col. 18, line 42-line 59) . Therefore, taking the combined disclosures of Fujiki and Lee as a whole, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate using the prior information, the problem can be redefined as aligning the projection of a 3D bounding box of the object into the back-projected rays of both sides of the 2D bounding box in a bird-eye view (BEV) of the object (i.e., from directly above the object. A unique pose hypotheses per sample point can be obtained by solving the equation. The RANSAC process (e.g., the algorithm shown in FIG. 7) or other suitable technique can be used to determine the optimal pose parameter with the maximum number of inliers among those hypotheses. In some cases, the determined pose can then be optimized as taught by Lee into the invention of Fujiki for the benefit of determining the optimal pose parameter with the maximum number of inliers among those hypotheses. (Lee: col. 18, lines 56-59) . Regarding claim 15 , the method of claim 14, Lee in the combination further disclose wherein adjusting a projected candidate region further comprises adjusting the projected candidate in a way to have the bottom of the projected candidate region above or aligned with the bottom of the bounding-box (col. 18, line 42-line 59) . Allowable Subject Matter 12-151-08 AIA 07-43 12-51-08 Claim 10 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims. 13-03-01 AIA The following is a statement of reasons for the indication of allowable subject matter: The prior art made of record and considered pertinent to the applicant's disclosure, taken individually or in combination, does not teach the claimed invention having the following limitations, in combination with the remaining claimed limitations. Regarding dependent claim 10 , the prior art does not teach or suggest the claimed invention having “wherein the final spatial position of the object is determined in the reference frame to be at distance (L+I)/pi from the approximate position of the object in a determined direction that corresponds to the upward vertical direction in the calibrated image, L and I being respectively the length and the width of the object and pi being the mathematical constant”, and a combination of other limitations thereof as recited in the claims . Conclusion 07-96 AIA The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Hayase et al., US 2010/0329344 discloses a scalable video encoding method of performing encoding by predicting an upper-layer signal having a relatively high spatial resolution by means of interpolation using an immediately-lower-layer signal having a relatively low spatial resolution. Zhang et al., US 8,855,929 discloses the initialization of an inertial navigation system is performed using information obtained from an image of an object. Dekel et al., US 2004/0002642 discloses a measurement system for tracking a pose of an object displaceable in a coordinate reference frame, such as a three dimensional coordinate system. Any inquiry concerning this communication or earlier communications from the examiner should be directed to VAN D HUYNH whose telephone number is (571)270-1937. The examiner can normally be reached 8AM-6PM. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Stephen R Koziol can be reached at (408) 918-7630. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. /VAN D HUYNH/Primary Examiner, Art Unit 2665 Application/Control Number: 18/749,436 Page 2 Art Unit: 2665 Application/Control Number: 18/749,436 Page 3 Art Unit: 2665 Application/Control Number: 18/749,436 Page 4 Art Unit: 2665 Application/Control Number: 18/749,436 Page 5 Art Unit: 2665 Application/Control Number: 18/749,436 Page 6 Art Unit: 2665 Application/Control Number: 18/749,436 Page 7 Art Unit: 2665 Application/Control Number: 18/749,436 Page 8 Art Unit: 2665 Application/Control Number: 18/749,436 Page 9 Art Unit: 2665 Application/Control Number: 18/749,436 Page 10 Art Unit: 2665 Application/Control Number: 18/749,436 Page 11 Art Unit: 2665 Application/Control Number: 18/749,436 Page 12 Art Unit: 2665 Application/Control Number: 18/749,436 Page 13 Art Unit: 2665 Application/Control Number: 18/749,436 Page 14 Art Unit: 2665 Application/Control Number: 18/749,436 Page 15 Art Unit: 2665 Application/Control Number: 18/749,436 Page 16 Art Unit: 2665
Prosecution Insights