DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 12/03/2025 has been entered.
Response to Arguments
Applicant’s arguments with respect to claim(s) 5-10, 12-18, 20-26 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 5-10, 12-18, 20-26 is/are rejected under 35 U.S.C. 103 as being unpatentable over Dargy et al. (US2015/0094885) in view of Paul et al. (WO2020/183345), Bansal (US2021/0041883) and Campbell (US12175577).
To claim 5, Dargy teach a method comprising:
receiving an image with image metadata, wherein the image metadata comprises an orientation and a position of a camera within three-dimensional space (paragraphs 0009, 0034, video images are captured with locations, which makes metadata obvious);
detecting an object and a location of the object within the image; determining, based on the location of the object within the image, the orientation of the camera, and the position of the camera, an estimated location of the object within the three-dimensional space; retrieving, for a plurality of known objects, a plurality of known locations within the three-dimensional space (Figs. 6-7; paragraphs 0032-0039);
comparing the plurality of known locations corresponding to the plurality of known objects to the estimated location of the object, wherein each of the plurality of known objects is equipped with a corresponding location transmitting device using which each corresponding known location is obtained (paragraphs 0012, 0034, 0037);
based on comparing the plurality of known locations corresponding to the plurality of known objects to the estimated location of the object, determining a known location of the plurality of known locations that matches the estimated location (paragraphs 0036, 0042, verify positional data in a PTC track data file for an asset, verification of a pre-recorded geographic location of an asset with an actual physical location associated with the asset); and
generating, for display, an indicator at the location of the object within the image, wherein the indicator comprises the known location of a known object within the three-dimensional space (Fig. 7; paragraphs 0041-0043, align an asset with position).
But, Dargy do not expressly disclose automatic comparing, wherein the known location is not identical to the estimated location, and wherein the known location of the known object within the three-dimensional space is different than the estimated location of the object within the image.
However, the known location is not identical to the estimated location in Dargy is obvious to one of ordinary skill in the art because a transverse distance from the rails to the asset is estimated (not exact) by one of the operators (paragraph 0042).
Similarly, Paul teach a method for surveying and updating asset (abstract), comprising:
receiving an image with image metadata, wherein the image metadata comprises an orientation and a position of a camera within three-dimensional space (page 3 lines 12-14, page 10 lines 27-30, page 19 lines 23-24, captured image data are digitized, time and data stamped and labelled with a GPS marker, obvious as metadata); detecting an object and a location of the object within the image; determining, based on the location of the object within the image, the orientation of the camera, and the position of the camera, an estimated location of the object within the three-dimensional space (Fig. 7; page 3 lines 26-29); retrieving, for a plurality of known objects, a plurality of known locations within the three-dimensional space (page 18); automatically comparing, using one or more processors, the plurality of known locations corresponding to the plurality of known objects to the estimated location of the object (page 2 lines 18-29, automating asset tracking and verifying location; page 7 lines 6-10, verification of an asset at a specified location; page 8 lines 12-15, system supports both automated and manual surveying of assets; page 11 lines 16-23, automated comparison with previously obtained data; page 23 lines 13-20, verify an existing object and confirm its exact location with respect to a closest GPS location in order to verify data on the database), wherein each of the plurality of known objects is equipped with a corresponding location transmitting device using which each corresponding known location is obtained (page 9 lines 17-19, page 10 lines 1-6, assets may be fitted with devices that report their location, condition and other traits that are tracked in real time); based on comparing the plurality of known locations corresponding to the plurality of known objects to the estimated location of the object, determining a known location of the plurality of known locations that matches the estimated location (paragraphs 0036, 0042, verify positional data in a PTC track data file for an asset, verification of a pre-recorded geographic location of an asset with an actual physical location associated with the asset); and generating, for display, an indicator at the location of the object within the image, wherein the indicator comprises the known location of a known object within the three-dimensional space (Figs. 1-2, previous survey and record; Figs. 3-4, current inspection and record; Fig. 5, verification record),
wherein the known location of the known object within the three-dimensional space is different than the estimated location of the object within the image (page 3 lines 26-29, page 5 lines 27-31, with LIDAR data using image data derived from street furniture assets whose pixel widths and/or or heights are compared with assumptions of their dimensions to estimate a location for the asset; page 23 lines 1-4, estimate target location in a 3D space; page 23 lines 13-20, identify or verify an existing object and confirm its exact location with respect to a closest GPS location in order to verify data on the database or initiate a query for an operator to verify later; which means estimated location of the object may not be identical to known location of the object in an older database).
In further Paul’s teaching, Bansal teach using GPS, vehicle camera, and LIDAR sensor to estimate location of object in 3D space in captured image (paragraphs 0019-0022, 0040, estimation would be different due to different cameras, different capturing position, etc.), wherein the estimated location having the lowest error score would be selected (paragraphs 0003-0005), such as the selected estimated location would be obviously not identical to the first estimated or recorded location (abstract, paragraph 0027), which correspond to Paul’s teaching in identifying, verifying and confirming an exact location of an object in later visits.
In furthering said obviousness, Campbell teach a system for asset recovery, roadway investigation, motor vehicle theft, locating a person of interest and/or providing roadway video for deep learning and autonomous driving training, wherein location metadata are captured along with video and objects in three-dimensional space in said video (abstract; Figs. 6, 11-14; column 34 line 49 to column 35 line 15).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate teaching of Paul, Bansal and Campbell into the method of Dargy, in order to further implementation of surveying and verifying through comparison of recorded asset location with estimated location thru captured image.
To claim 13, Dargy, Paul, Bansal and Campbell teach a non-transitory, computer-readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to perform operations (as explained in response to claim 1 above).
To claim 23, Dargy, Paul, Bansal and Campbell teach a system comprising: one or more processors; and a non-transitory, computer-readable storage medium storing instructions, which when executed by the one or more processors, cause the one or more processors to perform operations (as explained in response to claim 1 above).
To claims 6 and 14, Dargy, Paul, Bansal and Campbell teach claims 5 and 13.
Dargy, Paul, Bansal and Campbell teach wherein detecting the object and the location of the object within the image comprises: inputting the image into a machine learning model, wherein the machine learning model is trained to detect objects within images; and receiving, from the machine learning model, an object identifier of the object detected within the image and the location of the object within the image (Campbell, column 4 lines 41-65, column 23 lines 26-50, column 34 line 49 to column 35 line 15, column 37 lines 4-39)
To claims 7 and 15, Dargy, Paul, Bansal and Campbell teach claims 6 and 14.
Dargy, Paul, Bansal and Campbell teach further comprising: comparing the object identifier received from the machine learning model with object metadata associated with the known object; and based on the object identifier matching the object metadata, determining that the object detected within the image matches the known object (Dargy, paragraph 0010, the representation of the asset in the video image is aligned with the respective indicator, page 4 claim 1; Campbell, column 4 lines 19-65, column 23 lines 3-25, column 29 lines 46-59, column 36 lines 3-26).
To claims 8 and 16, Dargy, Paul, Bansal and Campbell teach claims 5 and 13.
Dargy, Paul, Bansal and Campbell teach wherein generating the indicator at the location of the object within the image comprises: determining based on metadata associated with the known object, a type associated with the object; retrieving an augmented reality identifier associated with the type; and generating for display the augmented reality identifier associated with the type at the location of the object within the image (Campbell, Figs. 6-7, column 15 lines 44-67, column 37 lines 4-39, column 45 line 64 to column 46 line 21).
To claims 9 and 17, Dargy, Paul, Bansal and Campbell teach claims 5 and 13.
Dargy, Paul, Bansal and Campbell teach wherein retrieving the plurality of known locations within the three-dimensional space comprises: transmitting a request onto a network for Global Positioning System (GPS) coordinates for the plurality of known objects; receiving, from the network, a plurality of GPS coordinates and a plurality of object identifiers associated with the plurality of GPS coordinates; and storing the plurality of GPS coordinates and the plurality of object identifiers (Dargy, paragraph 0025).
To claims 10 and 18, Dargy, Paul, Bansal and Campbell teach claims 9 and 17.
Dargy, Paul, Bansal and Campbell teach wherein determining the known location of the plurality of known locations that matches the estimated location comprises: comparing coordinates associated with the estimated location with the plurality of GPS coordinates; and determining a set of coordinates of the plurality of GPS coordinates that is closest to the coordinates associated with the estimated location, wherein the set of coordinates is associated with an object type matching the object type of the object within the image (Campbell, Figs. 6-7, column 15 lines 44-67, column 37 lines 4-39, column 45 line 64 to column 46 line 21).
To claims 12 and 20, Dargy, Paul, Bansal and Campbell teach claims 5 and 13.
Dargy, Paul, Bansal and Campbell teach further comprising: determining that a subset of objects within a plurality of objects detected within the image is of a same object type and that each object of the subset of objects is located within a threshold distance within the image of each other object of the subset of objects (Dargy, paragraph 0038); retrieving a corresponding unit identifier associated with each object of the subset of objects; determining that each object within the subset of objects has a matching unit identifier; and selecting the matching unit identifier as the group identifier (as explained in response to claim 11 above, wherein selecting matching unit identifier as group identifier would be an obvious practice by design preference).
To claim 21, Dargy, Paul, Bansal and Campbell teach claim 13.
Dargy, Paul, Bansal and Campbell teach wherein the instructions for retrieving the plurality of known locations within the three-dimensional space further cause the one or more processors to perform operations comprising: transmitting a request onto a network for coordinates for the plurality of known objects; receiving, from the network, a response comprising encrypted data; and decrypting the encrypted data to obtain a plurality of coordinates and a plurality of object identifiers associated with the plurality of coordinates (Campbell, column 30 lines 23-50).
To claim 22, Dargy, Paul, Bansal and Campbell teach claim 16.
Dargy, Paul, Bansal and Campbell teach wherein the instructions further cause the one or more processors to perform operations comprising applying corrections to the indicator within the image to stabilize the indicator on a representation of the object in the image (Dargy, paragraphs 0010, 0041, 0043, align, verify, record).
To claim 24, Dargy, Paul, Bansal and Campbell teach claim 23.
Dargy, Paul, Bansal and Campbell teach wherein the instructions for detecting the object and the location of the object within the image further cause the one or more processors to perform operations comprising: inputting the image into a machine learning model, wherein the machine learning model is trained to detect objects within images; and receiving, from the machine learning model, an object identifier of the object detected within the image and the location of the object within the image (as explained in response to claim 14 above).
To claims 25-26, Dargy, Paul, Bansal and Campbell teach claims 5 and 13.
Dargy, Paul, Bansal and Campbell teach further comprising: detecting, within the image, a plurality of objects and a plurality of locations corresponding to the plurality of objects; determining that a subset of objects within the plurality of objects each comprises metadata designating the subset of objects as a same type and that each object of the subject of objects is located within a threshold distance within the image of each other object of the subset of objects, and generating, for display, a group indicator for the subset of objects, wherein the group indicator comprises a group of identifier for the subset of objects and a corresponding location for each object within the subset of objects (as explained in responses to claims 11 and 19 above).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ZHIYU LU whose telephone number is (571)272-2837. The examiner can normally be reached Weekdays: 8:30AM - 5:00PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Stephen R Koziol can be reached at (408) 918-7630. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
ZHIYU . LU
Primary Examiner
Art Unit 2669
/ZHIYU LU/Primary Examiner, Art Unit 2665 January 27, 2026