Detailed Action
1. Claims 1-20 are pending in this Application.
Notice of Pre-AIA or AIA Status
2. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to amendment
3. Applicant’s response to the last Office Action filed on 02/19/2026 has been entered and made of record.
4. Claims1 and 16 have been amended.
.
Response to Argument
5. The Applicant’s argument filed 05/19/2026 is fully consider. For Examiner response see discussion below.
6. The Applicant’s has amended claim 1 as follows: “obtaining a query image from a depth-color image of a target object and performing object recognition on the query image to determine an object type of the target object; obtaining reference images corresponding to the query image by searching, among registered reference images, for have the same object type as the object type determined from
i. Regarding the limitation “obtaining a query image from a depth-color image of a target object” , Examiner respectfully disagrees with the Applicant’s argument because Hajime teach this limitation as follows:
obtaining a query image from a depth-color image of a target object ( Fig.2 Given a set of camera pose estimates for a query image) from a depth-color image of a target object (Fig.1,2, 3 and 5, the figures describe depth-color images ).
ii. However, it is noted that Hajime does not teach the underline section of the following limitation: “ performing object recognition on the query image to determine an object type of the target object; performing object recognition on the query image to determine an object type of the target object; obtaining reference images corresponding to the query image by searching, among registered reference images, for images having respective reference objects therein that have the same object type as the object type determined from the query image.”
Thus, the 35U.S.C 102 rejection based on Hajime is expressly withdraw.
After further search and consideration a new prior art (Tal Hassner et al., “Example Based 3D Reconstruction from Single 2D Image”) that teach the added limitation is found.
Regarding claim 16, it has been amended the same way as claim 1. Therefore, the arguments regarding claim 1 apply equally to claim 16.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103, which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
7. Claims 1-5 and 11-14 are rejected under 35 U.S.C. 103(a) as being unpatentable over Hajime et al., ( hereafter Hajime), “Is This the Right Place? Geometric-Semantic Pose Verification for Indoor Visual Localization” ICCV paper, IEEE explore, pub., 2019, in view of Tal Hassner et al., (hereafter Tal), “Example Based 3D Reconstruction from Single 2D Image”, Proceedings of the 2006 Conference on Computer Vision and Pattern Recognition Workshop.
As to claim 1, Hajime teaches A method performed by an electronic device, comprising:
obtaining a query image (Fig.2 Given a set of camera pose estimates for a query image) from a depth-color image of a target object (Fig.1,2, 3 and 5 ); and performing object recognition on the query image (page 4374 left right col. 1st par., Given database of geo-tagged images, place recognition approaches aim to identify the place depicted in a given query image, e.g., via image retrieval [3,19,49,69,76]. The geotag of the most similar database image is then often used to approximate the pose of the query image)
obtaining reference images corresponding to the query image by searching, among registered reference images (section 3.1 Candidate location retrieval. Inlock uses the NetVLAD [1] descriptor to identify the 100 database images most visually similar to the query.) wherein the reference images are obtained based on having respective reference objects therein that have a same object type as an object type of an object in the query image( as discussed above, the Inlock uses the NetVLAD [1] descriptor to identify the 100 database images most visually similar to the query image ) ;
determining a first semantic feature and first information corresponding to the query image, wherein the first information comprises first geometric information of the query image or first positional information of the query image; determining second semantic features and second pieces of information of the respectively corresponding reference images, wherein the second pieces of information each comprise second geometric information or second positional information of their respectively corresponding reference images, each reference image having a corresponding second semantic feature and second piece of information (Abstract, Section I left col., 2nd par., Fig. 1, Section 3 Geometric-Semantic Pose Verification, the authors propose multiple approaches for pose verification based on the combination of appearance, scene geometry, and semantic information. They integrate their approach into the Inlock pipeline [72], a state-of-the-art visual localization approach for large-scale indoor scenes. The approach verifies the estimated pose by comparing Geometric-Semantic Pose of query image and database images . Specifically the approach verifies the estimated pose by comparing the semantics and surface normal extracted from the query (d, j) and database (f, l) ); and
determining by a pose estimation neural network model (section 4, see the description of the Network architecture for pose verification.), a pose of the target object based on (i) the first semantic feature and the first information and (ii) the second semantic features and the second pieces of information ( as discussed above the approach verifies the estimated pose by comparing the semantics and surface normal extracted from the query (d, j) and database (f, l) );).
However, it is noted that Hajime does not specifically teaches the underline sections of the limitation: “performing object recognition on the query image to determine an object type of the target object, obtaining reference images corresponding to the query image by searching, among registered reference images, for images having respective reference objects therein that have the same object type as the object type determined from the query image”
On the other hand Tal teaches performing object recognition on the query image to determine an object type of the target object, obtaining reference images corresponding to the query image by searching, among registered reference images, for images having respective reference objects therein that (Fig. 1, page 2 section 3: Estimating depth from example mappings Given a query image I of some object of a certain class, our goal is to estimate a depth map D for the object. To determine depth our process uses examples of feasible mappings from intensities to depths for the class. These mappings are given in a database
PNG
media_image1.png
34
142
media_image1.png
Greyscale
PNG
media_image2.png
26
106
media_image2.png
Greyscale
where Ii and Di respectively are the image and the depth map of an object from the class. For simplicity we assume first that all the images in the database contain objects viewed in the same viewing position as in the query image. We relax this requirement later in Sec. 3.2. Our process attempts to associate a depth map D to the query image I, such that every patch of mappings in M = (I,D) will have a matching counterpart in database S. Thus, from the above discussion it is clear that searching for images by object type is like matching 3D image patches. Both methods break an image into small parts. They look for those parts in a database to find a match).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the object type matching method taught by Tal into the visual similarity search taught by Hajime, because doing so would improve search precision by removing false positives and creating a more robust image retrieval system.
As to claim 2, Hajime teaches the determining of the pose of the target object comprises: generating a first association feature of the query image based on the first semantic feature and the first geometric information of the query image (Abstract, Section I left col., 2nd par., Fig. 1, Geometric information and Semantic information is extracted from the query image QD, where the geometric information of QD, is surface normal extracted from the query Q (d, j));
generating a second association feature of the query image based on the second semantic features and the second pieces of geometric information of the reference images (Abstract, Section I left col., 2nd par., Fig. 1, similarly the Geometric information and Semantic information is extracted from the database image Q, where the geometric information of Q, is surface normal extracted from the query Q (f, l));and
determining the pose of the target object based on the first association feature and the second association feature(Abstract, Section I left col., 2nd par., Fig. 1, the estimated pose is calculated by comparing the semantics and surface normal extracted from the query (d, j) and database (f, l))
As to claim 3, Hajime teaches the obtaining of the reference images corresponding to the query image comprises: based on determining that the target object in the query image is an object registered in a database, obtaining the reference images from the database (section 3.1 right col., 2nd par., Candidate location retrieval. Inlock uses the NetVLAD [1] descriptor to identify the 100 database images most visually similar to the query).
As to claim 4, Hajime teaches the determining of the pose of the target object based on the first association feature and the second association feature comprises: generating correlation matrixes of correlation between the query image and each of the respectively corresponding reference images based on the first association feature and the second association feature, wherein each correlation matrix represents a relative position of a first pixel block of the query image with respect to a positionally-corresponding second pixel block of its corresponding reference image (Section 3.1 right col., 2nd par., InLoc’s dense pose verification stage then densely extracts RootSIFT [2,43] descriptors from both the synthetic and the real query image2. It then evaluates the (dis)similarity between the two images as the median of the inverse Euclidean distance between descriptors corresponding to the same pixel position. Let
PNG
media_image3.png
38
358
media_image3.png
Greyscale
be the local descriptor similarity function between Root- SIFT descriptors extracted at pixel position (x, y) in Q and QD. The similarity score between Q and QD); and
determining the pose of the target object based on the correlation matrixes ( as discuss above the pose is determined by evaluates the (dis)similarity between the two images as the median of the inverse Euclidean distance between descriptors corresponding to the same pixel position given by equation 1).
As to claim 5, Hajime teaches the generating of one of the correlation matrixes comprises: inputting the first association feature and the second association feature corresponding to the one of the correlation matrixes into an attention network(section 4. Trainable Pose Verification, Network architecture for pose verification. The network design follows an approach similar to that of DensePV, where given the original Q and a synthetic query image QD we first extract dense feature descriptors d(Q, x, y) and d(QD, x, y) using a fully convolutional network. Then, a descriptor similarity score map is computed by the cosine similarity
PNG
media_image4.png
32
324
media_image4.png
Greyscale
Finally, the 2D descriptor similarity score-map given by Eq. 7 is processed by a score regression CNN that estimates the agreement between Q and QD, resulting in a scalar
score.)
As to claim 11, Hajime teaches the determining of the pose of the target object comprises: selecting a target reference image from among the reference images based on a semantic feature corresponding to the query image, semantic features corresponding to each of the respective reference images, and similarity information associated with positional information between the query image and each of the reference images; and determining the pose of the target object based on the query image and the target reference image ( the limitations of these claim are discussed in claim 4 and 5 above, For example as discussed in claim 5 the Trainable Pose Verification Network architecture for pose verification design follows an approach similar to that of DensePV, where given the original Q and a synthetic query image QD the network first extract dense feature descriptors d(Q, x, y) and d(QD, x, y) using a fully convolutional network. Then, a descriptor similarity score map is computed by the cosine similarity
PNG
media_image4.png
32
324
media_image4.png
Greyscale
Finally, the 2D descriptor similarity score-map given by Eq. 7 is processed by a score regression CNN that estimates the agreement between Q and QD, resulting in a scalar
score.)
.
As to claim 12, Hajime teaches the determining of the target reference image from among the reference images comprises: for a first reference image of the reference images, determining a second pixel of the first reference image that is most similar to a first pixel of the query image from among pixels of the first reference image corresponding to a first position range with respect to the first pixel of the query image, based on the semantic feature of the query image and a semantic feature of the first reference image (Section 3.1 right col., 2nd par., InLoc’s dense pose verification stage then densely extracts RootSIFT [2,43] descriptors from both the synthetic and the real query image2. It then evaluates the (dis)similarity between the two images as the median of the inverse Euclidean distance between descriptors corresponding to the same pixel position. Let
PNG
media_image3.png
38
358
media_image3.png
Greyscale
be the local descriptor similarity function between Root- SIFT descriptors extracted at pixel position (x, y) in Q and QD. The similarity score between Q and QD););
for the first reference image, determining a third pixel of the first reference image that is most similar to the second pixel of the first reference image from among pixels of the query image corresponding to a second position range with respect to the second pixel of the first reference image, based on the semantic feature of the query image and the semantic feature of the first reference image; and determining the target reference image from among the reference images based on the first pixel, the second pixel, and the third pixel (Section 3.1 right col.2nd par., - page 4376 right col 1st par., .Euclidean distance between descriptors corresponding to the same pixel position given by equation 1
PNG
media_image3.png
38
358
media_image3.png
Greyscale
carry out similarity score between Q and QD pixel by pixel bases. The similarity score between Q and QD then is given by
PNG
media_image5.png
42
386
media_image5.png
Greyscale
The median is used instead of the mean as it is more robust to outliers. Invalid pixels, i.e., pixels into which no 3D point projects, are not considered in Eq. 2. Inlock finally selects the pose estimated using database image D that maximizes DensePV(Q,QD).)
As to claim 14, Hajime teaches the determining of the pose of the target object based on the query image and the target reference image comprises: generating a similarity matrix based on the first semantic feature of the query image and a second target semantic feature of the target reference image (Section 3.1 right col., 2nd par., Euclidean distance between descriptors corresponding to the same pixel position given by equation 1
PNG
media_image3.png
38
358
media_image3.png
Greyscale
carry out similarity score between Q and QD pixel by pixel bases);
optimizing the similarity matrix based on first saliency information of the query image, second target saliency information of the target reference image, first geometric consistency information of the query image, or second target geometric consistency information of the target reference image ( as discussed above the similarity score between Q and QD then is given by
PNG
media_image5.png
42
386
media_image5.png
Greyscale
The median is used instead of the mean as it is more robust to outliers. Invalid pixels, i.e., pixels into which no 3D point projects, are not considered in Eq. 2. Inlock finally selects the pose estimated using database image D that maximizes DensePV(Q,QD); and
determining the pose of the target object based on the optimized similarity matrix, a depth image corresponding to the query image, and a target depth image corresponding to the target reference image (Section 3.1 right col., 2nd par., The dense2D-2D matches between the query image and a retrieved database image define a set of 2D-3D matches when taking the depth map of the database image into account. The pose is then estimated using standard P3P-RANSAC [25].)
As to claim 13, Hajime teaches the determining of the target reference image from among the reference images based on the first pixel, the second pixel, and the third pixel comprises: for each reference image, determining a preset number of second pixel pairs from among first pixel pairs for a corresponding reference image, in order of similarity, wherein each of the first pixel pairs comprises the first pixel and the third pixel corresponding to the first pixel, and each of the second pixel pairs comprises the first pixel and the second pixel corresponding to the first pixel; fusing similarities of the second pixel pairs; and determining the target reference image from among the reference images, based on the fused similarity of the second pixel pairs for each reference image (Section 3.1 right col., 2nd par., as discussed above InLoc’s dense pose verification stage then densely extracts RootSIFT [2,43] descriptors from both the synthetic and the real query image2. It then evaluates the (dis)similarity between the two images as the median of the inverse Euclidean distance between descriptors corresponding to the same pixel position, where Euclidean distance given by equation
PNG
media_image3.png
38
358
media_image3.png
Greyscale
where the Euclidean distance equation(1) calculate (dis)similarity between the two images based on a pixel-by-pixel basis. Specifically equation (1) calculates the straight-line distance between corresponding pixel intensities (e.g., RGB or grayscale values) of two images. A lower distance indicates higher similarity. ).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103, which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
8. Claim 15-20 are rejected under 35 U.S.C. 103(a) as being unpatentable over
Hajime, “Is This the Right Place? Geometric-Semantic Pose Verification for Indoor Visual Localization ”, in view of Tal, “Example Based 3D Reconstruction from Single 2D Image”
further view of Simek et al., (hereafter Simek), US20180139431 A, Pub 05/17/2018
Regarding claim 15, while Hajime teaches the limitation of claim 1, fails to teach the limitation of calm 15.
On the other hand in the same field of endeavor a method of measuring similarity of images based on Euclidean distance of Simek teaches A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method of claim 1 (see claim 29)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the technique of storing a code that case the computer to excite the steps taught by Simek in order to store and execute the method claim 1 of Hajime.
The suggestion/motivation for doing so would have been to transfer the method of Hajime in remote locations using internet or storing in removable computer readable media, thus maximize electronically transferability and portability of the method taught by Hajime. Therefore, it would have been obvious to combine Simek with Hajime to obtain the invention as specified in claim 1.
As to claim 16, Simek teaches An electronic device, comprising: one or more processors; and a memory storing instructions configured to cause the one or more processors (see claim 29 and Fig. 12);
Regarding the remaining limitation of claim 16, all the claim limitations are set forth and rejected as per discussion for claim 1,.
Regarding claim 17, all the claim limitations are set forth and rejected as per discussion for claim 16 and 2.
As to claim18 the combination Hajime and Simek teaches the electronic device of claim 16, wherein the instructions are further configured to cause the one or more processors (Simek: claim 29 and Fig. 12);)to: based on determining that the target object in the query image is not registered in a database, obtain, as the reference images, images of the target object having respective poses through an image acquisition device ( Hajime, section 3.3, Projective Semantic Consistency PSC. Semantic consistency is then computed by counting the number of matching labels between the query and the synthetic image. In case of mismatch it would have been obvious to one ordinary skill in the art at time of filing to capture an image to overcome the inconsistence.).
Regarding claim 19, all the claim limitations are set forth and rejected as per discussion for claims 16 and 4.
Regarding claim 20, all the claim limitations are set forth and rejected as per discussion for claims 16 and 1.
Allowable Subject Matter
9. Claims 6-10 are objected to as being dependent upon a rejected base claims but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claim.
10. Regarding dependent claim 6 no prior art is found to anticipate or render the following limitation obvious:
“generating a first self-correlation feature of the query image and a second self-correlation feature of each of the reference images by inputting the first association feature and the second association feature into the first self-attention units, respectively; generating a first cross-correlation feature of the query image and a second cross-correlation feature of each of the reference images by inputting the first self-correlation feature and the second self-correlation feature into the first cross-attention unit; generating a third self-correlation feature of the query image and a fourth self-correlation feature of each of the reference images by inputting the first cross-correlation feature and the second cross-correlation feature into the second self-attention units, respectively; and generating the correlation matrix between the query image and each of the reference images based on the third self-correlation feature and the fourth self-correlation feature.”
11. Claims 7-10 are objected since they are depending on the objected claim 6.
12. Prior art not used in rejections but pertinent to the claims or disclosure
“Combining Depth, Color and Position Information for Object Instance Recognition on an Indoor Mobile Robot” to Louis-Charles Caron, Ph.D. Thesis Abstract, pub. 01/06/2016 disclosed:
Mobile robots have already entered people's homes to perform simple tasks for them. For robots at home to become real assistants, they have to be able to recognize the objects in their owner's home. In this thesis, object instance recognition algorithms are designed to cope with the variations (viewing angle, lighting conditions, etc.) that occur in the context of mobile robotics experiments. Several ways to take advantage of this context are studied. First, a geometric segmentation algorithm that benefits from the structure of indoor scenes to find isolated objects is designed. Then, a neural network based object recognition process that fuses shape, color and texture information provided by color and depth cameras is presented. This step highlights the importance of carefully combining multiple features and the difficulty to use color information in robotics. Finally, an alternative approach using a nearest neighbor classifier, which is easier to train than the neural network, is detailed. It relies on physical measures available to the robot to eliminate some parameters in clustering and nearest neighbor search procedures. Also, it uses information gathered through multiple sightings of the objects to reduce the negative impact of occlusions. The algorithm can recognize 52 objects with a success rate of 80\% and runs in 500 ms on average (on an Intel Core i5 CPU with 3~GB of RAM) on a mobile robot. This work shows that identifying and taking advantage of the structure and information available is essential to handle the variations that happen in indoor mobile robot experiments(see Abstract).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Contact Information
Any inquiry concerning this communication or earlier communication from the examiner should be directed to Mekonen Bekele whose telephone number is (469) 295-9077.The examiner can normally be reached on Monday -Friday from 9:00AM to 6:50 PM Eastern Time.
If attempt to reach the examiner by telephone are unsuccessful, the examiner’s supervisor Eng, George can be reached on (571) 272-7495.The fax phone number for the organization where the application or proceeding is assigned is 571-237-8300. Information regarding the status of an application may be obtained from the patent Application Information Retrieval (PAIR) system. Status information for published application may be obtained from either Private PAIR or Public PAIR.
Status information for unpublished application is available through Privet PAIR only.
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have question on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866.217-919 (tool-free)
/MEKONEN T BEKELE/Primary Examiner, Art Unit 2699
Prosecution Insights