DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Applicant’s claim for the benefit of a prior-filed application under 35 U.S.C. 119(e) or under 35 U.S.C. 120, 121, 365(c), or 386(c) is acknowledged.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 11/12/2024 and 04/10/2025 have been made record of and considered by the examiner.
Claim Objections
Claims 4 and 12 are objected to because of the following informalities: the claims recite the limitation “emphasizing a region.” “Emphasizing” fails to clarify what operation must occur, and the examiner is unsure if it is intended to mean highlighting, boxing, or something else. The specification fails to clarify the interpretation of the term. Appropriate correction is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-7, 9-15, and 17-20 are rejected under 35 U.S.C. 103 as being unpatentable over Kolagheichi-Ganjineh (US 2023/0129620 A1), and in further view of Schmalstieg (US 2014/0323148A1).
Consider claims 1, 9 and 17, Kolagheichi-Ganjineh discloses an electronic device, configured to construct a SLAM map with a SLAM module (¶132, 167, 189; “consistent local map representation is aggregated by using a first set of features for performing a 3D reconstruction under the assumption that all input images depict the same scenery from different points of view (i.e. a Structure from Motion assumption). This produces an aggregated local map, from the most recent images…”), comprising:
[claim 17: a non-transitory computer readable storage medium, wherein the non-transitory computer readable storage medium comprises one or more computer programs stored therein, and the one or more computer programs can be executed by one or more processors so as to be configured to operate a control method (¶133), wherein the control method comprises:]
[claim 9: a control method, suitable for an electronic device (¶133), comprising:]
a memory, configured to store a physical map and a database, wherein the database comprises a plurality of predefined common object images corresponding a plurality of common object 3D data (¶45; “These nodes may be "real" in that they represent a road intersection at which a minimum of three lines or segments intersect… In practically all modem digital maps, nodes and segments are further defined by various attributes which are again represented by data in the database.” ¶46-47; “the image content will typically include any objects that are included within the field of view of the camera or cameras at the point at which the image was recorded, so that the images capture not just information about the road and lane geometry, but also information regarding the general scenery, or environment, … e.g. including information about landmarks associated with the local environment such as buildings, traffic signs, traffic lights, billboards, etc., as well as information about the current conditions (and trafficability) of the road network.”; ¶133, 137-138, 159-165, 181, 215);
a camera circuit, configured to capture an environmental image (¶47, 167, 168; “at least one camera will obtain images of the scenery and road geometry ahead of the vehicle. However, one or more cameras may also be used to obtain images of the scenery surrounding the vehicle.); and
a processor, coupled to the memory and the camera circuit (¶33, 47, 167, 168), configured to:
obtain a first common object image from the environmental image (¶47, 167, 168; “at least one camera will obtain images of the scenery and road geometry ahead of the vehicle. However, one or more cameras may also be used to obtain images of the scenery surrounding the vehicle.”; input images; ¶185-188);
extract a plurality of feature points from the environmental image (¶169; “Next, this local map representation is used to embed and extract a secondary set of features, which are in turn used for matching, alignment and localisation purposes. This set of features may be comprised of but is not limited to: a 3D representation of the scenery's features (sparse point cloud); a 2D top-view map of detected landmarks and high-level features, such as lane markings, ground types or objects such as traffic lights, traffic signs, trees, sewer covers, etc. (orthomap);”);
adjust a plurality of first feature points of the feature points when a first common object 3D data of the plurality of common object 3D data is aligned to the first common object image (¶195-198, 214, 221; “The pixel contours of the masked cut-out may be vectorised to provide an accurate 3D reconstruction 1501 of the sign's shape, pixel content and position with respect to the vehicle's odometry, e.g. as shown in FIG. 15. This accurate reconstruction may be used for various follow up applications, such as visual global positioning of the type described herein, i.e. by associating the 3D sign reconstructions derived from a recording session with reference 3D signs in a map; or for refining monocular visual odometry and SLAM recording by allowing inaccuracies such as scale drift to be corrected by exploiting knowledge about real world sign sizes and using this information to normalize scale, e.g. in monocular structure from motion reconstructions.); and
update the SLAM map according to the plurality of feature points of the environmental image (¶155, 167, 249; “The landmark shapes, orientations and images output from the landmark observation creation and the lane geometry can thus be output to an observation datagram creation module for generating "datagrams" (or "roadagrams") that comprise localised map data, such as the lane and/or landmark observations described above, that has been extracted from the camera sensors, and that can be used e.g. for a localisation process and/or to update the HD map to more accurately reflect reality.”).
In related art, Schmalstieg explicitly discloses adjusting a plurality of first feature points of the feature points when a first common object 3D data of the plurality of common object 3D data is aligned to the first common object image (Schmalstieg ¶34; “the WAL Client may initialize the SLAM Map without first reading a prepopulated map. CAD model, markers in the scene, or other predefined descriptors from the WAL Server.”; ¶39-41; “The Global Localization response may represent a correction to a local map on the mobile device and can include rotation, translation, and scale information.”); and
update the SLAM map according to the plurality of feature points of the environmental image (Schmalstieg ¶39-44; updates to SLAM map; ¶63; “Upon receipt of the LR, the WAL Client can transform the SLAM map accordingly. The WAL Server may also send 3D points and 2D feature locations in the keyframe images. The 3D points and 2D feature locations can be used as constraints in the bundle adjustment process, to get a better alignment/correction of the SLAM map using non-linear refinement. This can be used to avoid drift (i.e., change in location over time) in the SLAM map.”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to incorporate the defined correction of Schmalstieg into the electronic mapping system of Kolagheichi-Ganjineh to yield the predictable outcome of improved accuracy. As stated by Schmalstieg, “This approach also enables incremental information from multiple viewpoints to produce enhanced localization accuracy (Schmalstieg ¶74).”
Consider claims 2, 10, and 18, Kolagheichi-Ganjineh, as modified by Schmalstieg, discloses the claimed invention wherein the first common object image matches a first predefined common object image of the plurality of predefined common object images, the first common object 3D data corresponds to the first predefined common object image, and the plurality of first feature points corresponds to the first common object image (Kolagheichi-Ganjineh ¶119-121, 158-167, 169-172, 221; Schmalstieg ¶34).
Consider claims 3 and 11, Kolagheichi-Ganjineh, as modified by Schmalstieg, discloses the claimed invention wherein the processor is further configured to: obtain a GPS position of the electronic device (Kolagheichi-Ganjineh ¶178, 180, 245; Schmalstieg ¶58); and
obtain the physical map and the database according to the GPS position (Kolagheichi-Ganjineh ¶159-166, 178, 180, 245; Schmalstieg ¶58).
Consider claims 4 and 12, Kolagheichi-Ganjineh, as modified by Schmalstieg, discloses the claimed invention wherein the database comprises a plurality of common object types (Kolagheichi-Ganjineh ¶45-47, 65, 133, 137-138, 159-165, 181, 215), the processor is further configured to: classify the first common object image into one of the plurality of common object types (Kolagheichi-Ganjineh ¶45-47, 59-69, 133, 137-138, 159-165, 181, 215); and
obtain and emphasize a region of the first common object image within the environmental image (Kolagheichi-Ganjineh ¶45-47, 59-66, 98, 149, 234).
Consider claims 5 and 13, Kolagheichi-Ganjineh, as modified by Schmalstieg, discloses the claimed invention wherein the processor is further configured to: abort the first common object image when the first common object 3D data is not aligned to the first common object image (Kolagheichi-Ganjineh ¶219; Schmalstieg ¶55, 62).
Consider claims 6, 14, and 19, Kolagheichi-Ganjineh, as modified by Schmalstieg, discloses the claimed invention wherein the processor is further configured to: adjust a size and a location of the first common object 3D data to align the first common object 3D data to the first common object image (Kolagheichi-Ganjineh ¶221; Schmalstieg ¶63);
obtain a plurality of second feature points according to the first common object 3D data when the first common object 3D data is aligned to the first common object image (Kolagheichi-Ganjineh ¶219-221; Schmalstieg ¶55-56); and
replace the plurality of first feature points with the plurality of second feature points (Kolagheichi-Ganjineh ¶219-221; Schmalstieg ¶56).
Consider claims 7, 15, and 20, Kolagheichi-Ganjineh, as modified by Schmalstieg, discloses the claimed invention wherein the processor is further configured to: update the SLAM map with a physical map information of the physical map when the environmental image is mapped to the physical map and the environmental image matches the SLAM map (Kolagheichi-Ganjineh ¶159-171; Schmalstieg ¶52).
Claims 8 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Kolagheichi-Ganjineh, in further view of Schmalstieg, as applied to claims 1-7, 9-15, and 17-20 above, and further in view of Ankenbauer (‘View-Invariant Localization using Semantic Objects in Changing Environments’).
Consider claims 8 and 16, Kolagheichi-Ganjineh, as modified by Schmalstieg, discloses the claimed invention wherein the environmental image further comprises a second common object image corresponding a second common object, wherein the first common object image corresponds to a first common object, wherein the processor is further configured to:
obtain a first map point of the first common object and a second map point of the second common object within the SLAM map (Kolagheichi-Ganjineh ¶26, 95-98, 119);
determine whether a distance between the first map point and the second map point fits a physical distance between a first physical object corresponding to the first common object and a second physical object corresponding to the second common object within the physical map (Kolagheichi-Ganjineh ¶26, 95-98, 119); and
determine that the environmental image is mapped to the physical map when the distance fits the physical distance (Kolagheichi-Ganjineh ¶26, 95-98, 119).
In related art, Ankenbauer explicitly discloses obtain a first map point of the first common object and a second map point of the second common object within the SLAM map (Ankenbauer Section III, A-E);
determine whether a distance between the first map point and the second map point fits a physical distance between a first physical object corresponding to the first common object and a second physical object corresponding to the second common object within the physical map (Ankenbauer Section III, A-E); and
determine that the environmental image is mapped to the physical map when the distance fits the physical distance (Ankenbauer Section III, A-F).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to incorporate the reference map and registration techniques of Ankenbauer into the electronic map alignment system of Kolagheichi-Ganjineh, as modified by Schmalstieg, to predictably yield improved matching validation.
Relevant Prior Art
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. US 2022/0237816 A1 discloses a method for displaying a virtual object, applied to an electrical device.
US 2018/0286065 A1 discloses methods for determining spatial coordinates of a 3D reconstruction.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ASHLEY HYTREK whose telephone number is (703)756-4562. The examiner can normally be reached M-F 9:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Steve Koziol can be reached at (408)918-7630. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ASHLEY HYTREK/Examiner, Art Unit 2665
/Stephen R Koziol/Supervisory Patent Examiner, Art Unit 2665
Prosecution Insights