DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments with respect to claim(s) 5-10, 12-18, 20-26 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 5-10, 12-18, 20-26 is/are rejected under 35 U.S.C. 103 as being unpatentable over Dargy et al. (US2015/0094885) in view of Madden (US2020/0410771), Paul et al. (WO2020/183345), and Campbell (US12175577).
To claim 5, Dargy teach a method comprising:
receiving an image with image metadata, wherein the image metadata comprises an orientation and a position of a camera within three-dimensional space (paragraphs 0009, 0034, video images are captured with locations, which makes metadata obvious);
detecting an object and a location of the object within the image; determining, based on the location of the object within the image, the orientation of the camera, and the position of the camera, an estimated location of the object within the three-dimensional space; retrieving, for a plurality of known objects, a plurality of known locations within the three-dimensional space (Figs. 6-7; paragraphs 0032-0039);
comparing, using one or more processors, a plurality of sets of coordinates (GPS coordinates) corresponding to the plurality of known locations of the plurality of known objects to an estimated set of coordinates corresponding to the estimated location of the object, wherein each of the plurality of known objects is equipped with a corresponding location transmitting device using which each corresponding known location is obtained (paragraphs 0012, 0034, 0037), and wherein the estimated set of coordinates is compared to multiple sets of coordinates (obvious in scenario of having multiple recorded assets with corresponding coordinates);
based on comparing the plurality of known locations corresponding to the plurality of known objects to the estimated location of the object, determining a known location of the plurality of known locations that matches the estimated location (paragraphs 0036, 0042, verify positional data in a PTC track data file for an asset, verification of a pre-recorded geographic location of an asset with an actual physical location associated with the asset); and
generating, for display, an indicator at the location of the object within the image, wherein the indicator comprises the known location of a known object within the three-dimensional space (Fig. 7; paragraphs 0041-0043, align an asset with position).
But, Dargy do not expressly disclose automatic comparing, wherein the known location is not identical to the estimated location, generating an overlay of one or more pixels on top of the image as visual indicator at the estimated location of the object within the image, and wherein the known location of the known object within the three-dimensional space is different than the estimated location of the object within the image.
However, the known location is not identical to the estimated location in Dargy is obvious to one of ordinary skill in the art because a transverse distance from the rails to the asset is estimated (not exact) by one of the operators (paragraph 0042).
In furthering said obviousness in Dargy, Madden teach obtaining an live image of an area from augmented reality device and identifying the area based on the image obtained from the augmented reality device (paragraphs 0004-0005, 0020), wherein said identifying comprises comparing determined coordinate to a plurality of coordinates corresponding to portions of a property (paragraph 0080), wherein said plurality of coordinates are known and stored (paragraphs 0078, 0081-0085), which would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate into the method of Dargy, in order to implement detail of comparison for coordinate identification.
Similarly to Dargy and instant application, Paul teach a method for surveying and updating asset (abstract), comprising: receiving an image with image metadata, wherein the image metadata comprises an orientation and a position of a camera within three-dimensional space (page 3 lines 12-14, page 10 lines 27-30, page 19 lines 23-24, captured image data are digitized, time and data stamped and labelled with a GPS marker, obvious as metadata); detecting an object and a location of the object within the image; determining, based on the location of the object within the image, the orientation of the camera, and the position of the camera, an estimated location of the object within the three-dimensional space (Fig. 7; page 3 lines 26-29); retrieving, for a plurality of known objects, a plurality of known locations within the three-dimensional space (page 18); automatically comparing, using one or more processors, the plurality of known locations corresponding to the plurality of known objects to the estimated location of the object (page 2 lines 18-29, automating asset tracking and verifying location; page 7 lines 6-10, verification of an asset at a specified location; page 8 lines 12-15, system supports both automated and manual surveying of assets; page 11 lines 16-23, automated comparison with previously obtained data; page 23 lines 13-20, verify an existing object and confirm its exact location with respect to a closest GPS location in order to verify data on the database), wherein each of the plurality of known objects is equipped with a corresponding location transmitting device using which each corresponding known location is obtained (page 9 lines 17-19, page 10 lines 1-6, assets may be fitted with devices that report their location, condition and other traits that are tracked in real time); based on comparing the plurality of known locations corresponding to the plurality of known objects to the estimated location of the object, determining a known location of the plurality of known locations that matches the estimated location (paragraphs 0036, 0042, verify positional data in a PTC track data file for an asset, verification of a pre-recorded geographic location of an asset with an actual physical location associated with the asset); and generating, for display, an indicator at the location of the object within the image, wherein the indicator comprises the known location of a known object within the three-dimensional space (Figs. 1-2, previous survey and record; Figs. 3-4, current inspection and record; Fig. 5, verification record),
wherein the known location of the known object within the three-dimensional space is different than the estimated location of the object within the image (page 3 lines 26-29, page 5 lines 27-31, with LIDAR data using image data derived from street furniture assets whose pixel widths and/or or heights are compared with assumptions of their dimensions to estimate a location for the asset; page 23 lines 1-4, estimate target location in a 3D space; page 23 lines 13-20, identify or verify an existing object and confirm its exact location with respect to a closest GPS location in order to verify data on the database or initiate a query for an operator to verify later; which means estimated location of the object may not be identical to known location of the object in an older database).
In furthering said obviousness, Campbell teach a system for asset recovery, roadway investigation, motor vehicle theft, locating a person of interest and/or providing roadway video for deep learning and autonomous driving training, which includes generating for display, as an overlay of one or more pixels on top of the image, a visual indicator at the estimated location of the object within the image (status caption 412a-f of Fig. 6; column 36 lines 3-60, object identification with granular location coordinate and distance from the camera), wherein location metadata are captured along with video and objects in three-dimensional space in said video (abstract; Figs. 6, 11-14; column 34 line 49 to column 35 line 15).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate teachings of Paul and Campbell into the method of Dargy and Madden, in order to further implementation of surveying and verifying through comparison of recorded asset location with estimated location thru captured image/video.
To claim 13, Dargy, Madden, Paul and Campbell teach a non-transitory, computer-readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to perform operations (as explained in response to claim 1 above).
To claim 23, Dargy, Madden, Paul and Campbell teach a system comprising: one or more processors; and a non-transitory, computer-readable storage medium storing instructions, which when executed by the one or more processors, cause the one or more processors to perform operations (as explained in response to claim 1 above).
To claims 6 and 14, Dargy, Madden, Paul and Campbell teach claims 5 and 13.
Dargy, Madden, Paul and Campbell teach wherein detecting the object and the location of the object within the image comprises: inputting the image into a machine learning model, wherein the machine learning model is trained to detect objects within images; and receiving, from the machine learning model, an object identifier of the object detected within the image and the location of the object within the image (Campbell, column 4 lines 41-65, column 23 lines 26-50, column 34 line 49 to column 35 line 15, column 37 lines 4-39)
To claims 7 and 15, Dargy, Madden, Paul and Campbell teach claims 6 and 14.
Dargy, Madden, Paul and Campbell teach further comprising: comparing the object identifier received from the machine learning model with object metadata associated with the known object; and based on the object identifier matching the object metadata, determining that the object detected within the image matches the known object (Dargy, paragraph 0010, the representation of the asset in the video image is aligned with the respective indicator, page 4 claim 1; Campbell, column 4 lines 19-65, column 23 lines 3-25, column 29 lines 46-59, column 36 lines 3-26).
To claims 8 and 16, Dargy, Madden, Paul and Campbell teach claims 5 and 13.
Dargy, Madden, Paul and Campbell teach wherein generating the indicator at the location of the object within the image comprises: determining based on metadata associated with the known object, a type associated with the object; retrieving an augmented reality identifier associated with the type; and generating for display the augmented reality identifier associated with the type at the location of the object within the image (Campbell, Figs. 6-7, column 15 lines 44-67, column 37 lines 4-39, column 45 line 64 to column 46 line 21).
To claims 9 and 17, Dargy, Madden, Paul and Campbell teach claims 5 and 13.
Dargy, Madden, Paul and Campbell teach wherein retrieving the plurality of known locations within the three-dimensional space comprises: transmitting a request onto a network for Global Positioning System (GPS) coordinates for the plurality of known objects; receiving, from the network, a plurality of GPS coordinates and a plurality of object identifiers associated with the plurality of GPS coordinates; and storing the plurality of GPS coordinates and the plurality of object identifiers (Dargy, paragraph 0025).
To claims 10 and 18, Dargy, Madden, Paul and Campbell teach claims 9 and 17.
Dargy, Madden, Paul and Campbell teach wherein determining the known location of the plurality of known locations that matches the estimated location comprises: comparing coordinates associated with the estimated location with the plurality of GPS coordinates; and determining a set of coordinates of the plurality of GPS coordinates that is closest to the coordinates associated with the estimated location, wherein the set of coordinates is associated with an object type matching the object type of the object within the image (Campbell, Figs. 6-7, column 15 lines 44-67, column 37 lines 4-39, column 45 line 64 to column 46 line 21).
To claims 12 and 20, Dargy, Madden, Paul and Campbell teach claims 5 and 13.
Dargy, Madden, Paul and Campbell teach further comprising: determining that a subset of objects within a plurality of objects detected within the image is of a same object type and that each object of the subset of objects is located within a threshold distance within the image of each other object of the subset of objects (Dargy, paragraph 0038); retrieving a corresponding unit identifier associated with each object of the subset of objects; determining that each object within the subset of objects has a matching unit identifier; and selecting the matching unit identifier as the group identifier (as explained in response to claim 11 above, wherein selecting matching unit identifier as group identifier would be an obvious practice by design preference).
To claim 21, Dargy, Madden, Paul and Campbell teach claim 13.
Dargy, Madden, Paul and Campbell teach wherein the instructions for retrieving the plurality of known locations within the three-dimensional space further cause the one or more processors to perform operations comprising: transmitting a request onto a network for coordinates for the plurality of known objects; receiving, from the network, a response comprising encrypted data; and decrypting the encrypted data to obtain a plurality of coordinates and a plurality of object identifiers associated with the plurality of coordinates (Campbell, column 30 lines 23-50).
To claim 22, Dargy, Madden, Paul and Campbell teach claim 16.
Dargy, Madden, Paul and Campbell teach wherein the instructions further cause the one or more processors to perform operations comprising applying corrections to the indicator within the image to stabilize the indicator on a representation of the object in the image (Dargy, paragraphs 0010, 0041, 0043, align, verify, record).
To claim 24, Dargy, Madden, Paul and Campbell teach claim 23.
Dargy, Madden, Paul and Campbell teach wherein the instructions for detecting the object and the location of the object within the image further cause the one or more processors to perform operations comprising: inputting the image into a machine learning model, wherein the machine learning model is trained to detect objects within images; and receiving, from the machine learning model, an object identifier of the object detected within the image and the location of the object within the image (as explained in response to claim 14 above).
To claims 25-26, Dargy, Madden, Paul and Campbell teach claims 5 and 13.
Dargy, Madden, Paul and Campbell teach further comprising: detecting, within the image, a plurality of objects and a plurality of locations corresponding to the plurality of objects; determining that a subset of objects within the plurality of objects each comprises metadata designating the subset of objects as a same type and that each object of the subject of objects is located within a threshold distance within the image of each other object of the subset of objects, and generating, for display, a group indicator for the subset of objects, wherein the group indicator comprises a group of identifier for the subset of objects and a corresponding location for each object within the subset of objects (as explained in responses to claims 11 and 19 above).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ZHIYU LU whose telephone number is (571)272-2837. The examiner can normally be reached Weekdays: 8:30AM - 5:00PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Stephen R Koziol can be reached at (408) 918-7630. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
ZHIYU . LU
Primary Examiner
Art Unit 2669
/ZHIYU LU/Primary Examiner, Art Unit 2665 August 28, 2026