Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Specification
The disclosure is objected to because of the following informalities:
Page 4, Paragraph 17, Line 18, there is an “e” at the end of the paragraph that should be removed so that it then reads as “... according to an embodiment of the invention. [[e]]”
Page 10, Paragraph 45, Line 28, there is an extra period “.” after “invention” that should be removed so that it then reads as “... certain embodiments of the invention. [[.]]”
Page 15, Paragraph 60, Line 12, the word “form” should instead be “from”, so that it then reads as “...from the information processing system 230, and/or from a plurality of eternal systems.”
Page 16, Paragraph 67, Line 30, there is a “1” before “three-dimensional building model” that should not be there and instead should maybe be replaced with “, the”, so that it then reads as “... according to certain embodiments of the present invention[[. 1]], the three-dimensional building model information generated ...”
Appropriate correction is required.
Claim Objections
Claim 7 is objected to because of the following informalities: On Page 2, Line 24, there is a “:” missing after the word “comprises” and should read as “...view image using the binary mask comprises: ...”.
Claim 11 is objected to because of the following informalities: On Page 3, Line 26, the word “struct” should be “structure” and should then read as “... as the same road traffic structure, ...”.
Claim 15 is objected to because of the following informalities: On Page 4, Line 20, the abbreviation of building control point should be BCP, instead of GCP, and should then read as “...wherein the map matching point comprises a ground control point (GCP) and a building control point (BCP) ...”.
Appropriate correction is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 6, 9-14 and 16-19 are rejected under 35 U.S.C. 103 as being unpatentable over Ding et al. (U.S. Patent: #12,651,373 B2), hereinafter Ding, in view of Chen et al. (Pub. No.: US 2018/0322654 A1), hereinafter Chen, and further in view of Chiu et al. (Pub. No.: US 2020/0357143 A1).
Regarding claim 1, Ding discloses a method for generating a visual feature map using a three-dimensional model and a street view image, performed by at least one processor (Col. 1, Lines 65-67 teach that embodiments of this application provide a visual localization method, to implement localization with high precision when an image photographing scenario changes greatly.), the method comprising:
receiving a three-dimensional model for a specific area including three-dimensional geometric information expressed as absolute coordinate positions (FIG. 10 and Col. 24, Lines 11-18 teach that the LOD model is generated based on the satellite image, a road layer is extracted, and candidate points are selected at spacings on the road, for example, at a spacing of one meter. A specific spacing value is not limited herein. FIG. 10 is a schematic diagram of an embodiment of selecting a candidate point according to the embodiments of this application. A point on a road in the figure is a candidate point.);
receiving a first street view image captured within the specific area (Col. 26, Lines 44-55 teach that the terminal provided in this embodiment of this application includes: an obtaining unit 1501, configured to obtain an image of a building; and a generation unit 1502, configured to generate a descriptor based on the image, where the descriptor includes information about a horizontal viewing angle between a first vertical feature line and a second vertical feature line in the image, the first vertical feature line indicates a first facade intersection line of the building, and the second vertical feature line indicates a second facade intersection line of the building.). However, Ding fails to disclose a first street view image captured at a first node.
Chen discloses a first street view image captured at a first node (Paragraph 38 teaches that at step 204, the mobile device obtains a set of street view images. For example, the street view images may be taken by a vehicle-mounted camera along a street and are stored in one or more databases in a computing network. Additionally, paragraph 40 teaches that at step 206, the mobile device obtains a first graph of the image and a plurality of second graphs of the set of street view images. For example, the first graph and the plurality of second graphs may be generated by the mobile device locally through image recognition. Alternatively, the first graph and the plurality of second graphs may be obtained from a remote computer system or server in a computing network. Each of the first graph and the plurality of second graphs includes a plurality of nodes and a plurality of edges between nodes, and each node represents a specific image point such as an interest point.). Since Ding teaches a visual localization method for generating three-dimensional models based on images of buildings obtained at street view level and Chen teaches obtaining street view images with node associations, it would have been obvious to a person having ordinary skill in the art to combine the two features together so that any of the images obtained at street level would then have a corresponding node related to a certain image and/or feature.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Ding to incorporate the teachings of Chen, so that the combined features together would improve the understanding of specific features and objects within the images by providing the capabilities to utilize node specific relationships.
Furthermore, Ding in view of Chen disclose projecting at least some of the three-dimensional geometric information included in the three-dimensional model onto the first street view image, based on absolute coordinate position information and direction information of the first street view image (Col. 18, Lines 10-13 of Ding teach that a cylindrically projected image is projected onto a plane xOz on which the x-axis and z-axis of the camera coordinate system are located, to obtain a circle with an optical center O as a circle center. Additionally, Col. 22 Lines 1-6 teach that the azimuth information is angle information obtained by the terminal sensor when the image is photographed. Optionally, the azimuth information is an orientation angle obtained by a magnetometer, represents an orientation that is present when the image is photographed, and is specifically a direction in an east-north-up (ENU) coordinate system.) However, Ding in view of Chen fail to disclose to render a depth map associated with the first street view image.
Chiu discloses to render a depth map associated with the first street view image (Paragraph 102 teaches that for harvesting LIDAR Depth Images, aerial LIDAR point-cloud of the selected area was collected to create a DEM which is rendered exhaustively from multiple locations and viewpoints. For each GPS location containing RGB images, the LIDAR depth images was rendered from 1.7 m above ground and paragraph 93 teaches that the feature vector of an RGB image and a LIDAR depth image are denoted as g (∈custom-character.sup.I) and (∈custom-character.sup.L), respectively. A linear projection can be used to project both modalities in a common space according to r.sub.p=W.sup.(r)g (r.sub.p∈custom-character.sup.J) and d.sub.p=W.sup.(d)a (d.sub.p∈custom-character.sup.J) where the transformation matrices for the RGB and LIDAR depth maps are W.sup.i∈custom-character.sup.J×I and W.sup.d∈custom-character.sup.J×L, respectively.). Since Ding in view of Chen teach a function for projecting three-dimensional geometric information with relation to coordinates and directions of a street view image and Chiu teaches the capabilities of using and generating depth map images in relation to different locations and viewpoints, including those at street level, it would have been obvious to a person having ordinary skill in the art to combine the two features together so that the geometric, position and directional information could be utilized together to process and render a depth map associated with a street view image.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Ding in view of Chen to incorporate the teachings of Chui, so that the combined features together would allow for depth maps to be able to be created using geometric, positional and directional information, which would improve the overall precision, accuracy and recognition of mapping features related to the street view images.
Furthermore, Ding in view of Chen and Chui disclose extracting a first set of feature points from the first street view image (Col. 16, Lines 57-58 of Ding teach that 1002: Extract a vertical feature line of the building from the image and Col. 18, Lines 13-16 of Ding teach that a vertical feature line of the building obtained from the image is projected into a feature point on the circle. The feature point represents a vertical feature line of the building obtained from the image. Additionally, paragraph 34 of Chiu teaches that the visual feature extraction module 110 comprises a convolutional neural network (CNN) trained to extract visual features from an input image stream.);
determining absolute coordinate position information of at least some of the first set of feature points, based on the depth map (Col. 21, Lines 45-55 of Ding teach that the first localization information includes localization information obtained based on a GPS signal, a Wi-Fi signal, location information of a base station serving the terminal, or localization information obtained by manually entering an address by a user. The localization information has low precision, and may provide reference for visual localization. Within a first location range determined based on the first localization information, visual localization that is more accurate than the first localization information is further obtained in a geometric semantic descriptor matching manner. Also, Col. 20, Lines 22-27 of Ding teach that optionally, a first feature point and a second feature point on a circle that is on an xOz plane of a camera coordinate system and that uses an optical center O as a circle center are obtained by performing conversion to a cylindrical projection on the first vertical feature line and the second vertical feature line of the building that are extracted from the image. Lastly, paragraph 84 of Chiu teaches that in some embodiments in accordance with the present principles, segmentations maps (e.g., semantic segmentation maps) from the paired RGB image can be used as the ground-truth maps to train the segmentation network for the other modalities, such as the LIDAR depth maps.);
and generating a first visual feature map associated with the first street view image by storing, for each of the feature points in the first set, the absolute coordinate position information and a visual feature descriptor in association with each other (Paragraph 34 of Chiu teaches that the output of the visual feature extraction module 110 includes visual feature maps. Additionally, Col. 11, Lines 51-59 of Ding teach that in a possible implementation of the sixth aspect, the generation unit is specifically configured to: extract the first vertical feature line and the second vertical feature line of the building from the image; and generate the descriptor based on a location relationship between the first vertical feature line and the second vertical feature line, where the location relationship includes the information about the horizontal viewing angle between the first vertical feature line and the second vertical feature line.).
Regarding claim 6, Ding in view of Chen and Chiu disclose everything claimed as applied above (see claim 1), in addition, Ding in view of Chen and Chiu disclose wherein the three-dimensional model comprises a plurality of three- dimensional building models and road models within the specific area (Col. 5, Lines 35-41 of Ding teach that according to the visual localization method provided in this embodiment of this application, the terminal may widen a field of view by photographing a plurality of images, to obtain image information of more buildings. The first image and the second image that include partially overlapping image information may be stitched, and are used to generate the descriptor.),
wherein the depth map comprises depth information of buildings and the roads (Col. 3, Lines 63-67 of Ding teach that in a possible implementation of the first aspect, the first data includes the height information and/or the depth information of the first vertical feature line, and the second data includes the height information and/or the depth information of the second vertical feature line.),
and wherein absolute coordinate position information of feature points associated with buildings and roads, among the first set of feature points, is determined based on the depth map (Col. 20, Lines 22-37 of Ding teach that optionally, a first feature point and a second feature point on a circle that is on an xOz plane of a camera coordinate system and that uses an optical center O as a circle center are obtained by performing conversion to a cylindrical projection on the first vertical feature line and the second vertical feature line of the building that are extracted from the image. The descriptor includes angle information corresponding to an arc including the first feature point and the second feature point. Optionally, the geometric semantic descriptor includes only angle information between feature points. Optionally, height information of a vertical contour line of the building on the image may alternatively be recorded in information corresponding to feature points, and the geometric semantic descriptor includes angle information between feature points and height information of the feature points. Additionally, Col. 20, Lines 39-50 of Ding teach that optionally, if a sensor has depth data or obtains a depth value of a vertical contour line through binocular depth estimation or monocular depth recovery, or by using another technology, the depth data or the depth value of the vertical contour line may alternatively be recorded in information corresponding to feature points. The geometric semantic descriptor includes angle information between feature points and depth information of the feature points. The height information is used to indicate a height of the second facade intersection line, and the depth information is used to indicate a distance from a photographing place of the image to a facade intersection line of the corresponding building.).
Regarding claim 9, Ding in view of Chen and Chiu disclose everything claimed as applied above (see claim 1), in addition, Ding in view of Chen and Chiu disclose receiving a second street view image captured at a second node within the specific area (Paragraph 38 of Chen teaches that at step 204, the mobile device obtains a set of street view images. For example, the street view images may be taken by a vehicle-mounted camera along a street and are stored in one or more databases in a computing network and paragraph 40 of Chen teaches that at step 206, the mobile device obtains a first graph of the image and a plurality of second graphs of the set of street view images. For example, the first graph and the plurality of second graphs may be generated by the mobile device locally through image recognition. Alternatively, the first graph and the plurality of second graphs may be obtained from a remote computer system or server in a computing network. Each of the first graph and the plurality of second graphs includes a plurality of nodes and a plurality of edges between nodes, and each node represents a specific image point such as an interest point.);
generating three-dimensional planar information for a specific road traffic structure included in the first street view image and the second street view image, based on the first street view image and the second street view image (Paragraph 40 of Chen teaches that each of the first graph and the plurality of second graphs includes a plurality of nodes and a plurality of edges between nodes, and each node represents a specific image point such as an interest point. In some embodiments, the nodes may represent detectable objects in an image, such as trees, buildings, roads.);
and determining absolute coordinate position information of feature points associated with the specific road traffic structure, among the first set of feature points, based on the three-dimensional planar information (Paragraph 59 of Chen teaches that FIG. 6B illustrates example street view image 600 (shown in FIG. 6A) tagged with interest points, in accordance with one embodiment of the present invention. In image 630, some interest points are identified, for example, interest points 631, 632, 633, and 634 and paragraph 61 of Chen teaches that according to embodiments of the present disclosure, since the part of nodes 561-571 in graph 560 and the part of nodes 661-671 in graph 660 are similar, image 500 and image 600 may be determined to be matched with each other. According to embodiments of the present disclosure, the matching process between two graphs may be carried out as the partial graph matching; for example, a standard partial graph matching process with a specific edge value representation. In this way, the location of the mobile device can be determined accurately by partly matching two graphs generated from the images. In addition, Col. 22, Lines 28-38 of Ding teach that matching is performed between the geometric semantic descriptor of the image and a descriptor of a candidate point in the preset descriptor database, a candidate point related to the localization information is obtained based on a similarity obtained through matching, and visual localization of a photographing place of the image may be determined based on a coordinate location of the candidate point on the map. That is, a geographical location of a candidate point indicated by a descriptor matching the descriptor of the image in the preset descriptor database is localization information of the photographing place of the image.).
Regarding claim 10, Ding in view of Chen and Chiu disclose everything claimed as applied above (see claim 9), in addition, Ding in view of Chen and Chiu disclose wherein the determining of the absolute coordinate position information of feature points associated with the specific road traffic structure comprises:
projecting the feature points associated with the specific road traffic structure, among the first set of feature points, onto the three-dimensional plane (FIG. 5B and paragraph 56 of Chen teaches that FIG. 5B illustrates the captured example image 500 (shown in FIG. 5A) tagged with interest points, in accordance with one embodiment of the present invention. In tagged image 530, some interest points are identified, for example, interest points 531, 532, 533 and 534. It should to be understood, to avoid confusion of illustration, only some interest points are tagged with numerals.).
Regarding claim 11, Ding in view of Chen and Chiu disclose everything claimed as applied above (see claim 9), in addition, Ding in view of Chen and Chiu disclose wherein the generating of the three-dimensional planar information comprises:
detecting a first area including a first road traffic structure from the first street view image (Paragraph 43 of Chen teaches that the set of street view images may be obtained based on the geographic information from the mobile device. For example, the street view images within the area determined from the geographic information are selected to match the captured image. Accordingly, only the street view images near the area where the mobile device is located are obtained, thereby reducing the number of street view images to be matched. Additionally, paragraph 40 of Chen teaches that in some embodiments, the nodes may represent detectable objects in an image, such as trees, buildings, roads.);
detecting a second area including a second road traffic structure from the second street view image (Paragraph 102 of Chiu teaches that for harvesting LIDAR Depth Images, aerial LIDAR point-cloud of the selected area was collected to create a DEM which is rendered exhaustively from multiple locations and viewpoints. For each GPS location containing RGB images, the LIDAR depth images was rendered from 1.7 m above ground. Additionally, paragraph 40 of Chen teaches that in some embodiments, the nodes may represent detectable objects in an image, such as trees, buildings, roads.);
determining the first road traffic structure and the second road traffic structure to be the specific road traffic structure, as the same road traffic struct, based on visual similarity between the first area and the second area (FIG. 4 and paragraph 68 of Chiu teach that in FIG. 4, in the third and fourth rows (FIG. 4b), a query image from a winter sequence of the Nordland dataset is matched against retrieved database images from a summer sequence of the Nordland dataset. In FIG. 4b, a method relying on combined appearance and semantic features without attention, retrieves an image that is very similar to the query image, however the retrieved image lacks the distinctive fence structure on the left of the image.);
and performing stereo matching and triangulation on the first area and the second area to generate three-dimensional planar information for the specific road traffic structure (Paragraph 67 of Chiu teaches that FIG. 4 depicts how, across time of day, the maps from both attention modalities remain consistent and focus on stable features, with the appearance attention focusing mostly on fine details, such as lane markings or architectural elements, and the semantic attention giving emphasis to scene layout and skyline shape. In FIG. 4, although the appearance modality attends highly to a bus's features in the matched image, as if it were any other structure, the semantic attention module has learned to disregard the region containing the bus as a dynamic object and gives more emphasis to the remaining scene layout. Additionally, paragraph 68 of Chiu teaches that in the embodiment of FIG. 4b, an image retrieved using the visual localization system in accordance with the present principles retrieves an image that is an exact location match of the query image.).
Regarding claim 12, Ding in view of Chen and Chiu disclose everything claimed as applied above (see claim 11), in addition, Ding in view of Chen and Chiu disclose wherein the visual similarity between the first area and the second area is determined using at least one of color similarity, visual feature descriptor similarity, or a deep learning-based matching model (Col. 22, Lines 28-38 of Ding teach that matching is performed between the geometric semantic descriptor of the image and a descriptor of a candidate point in the preset descriptor database, a candidate point related to the localization information is obtained based on a similarity obtained through matching, and visual localization of a photographing place of the image may be determined based on a coordinate location of the candidate point on the map. That is, a geographical location of a candidate point indicated by a descriptor matching the descriptor of the image in the preset descriptor database is localization information of the photographing place of the image.).
Regarding claim 13, Ding in view of Chen and Chiu disclose everything claimed as applied above (see claim 1), in addition, Ding in view of Chen and Chiu disclose wherein the extracting of the first set of feature points comprises:
converting the first street view image into a plurality of planar images using a perspective projection method (Col. 19, Lines 7-10 of Ding teach that an entire centrally projected image that is obtained needs to be converted into a cylindrically projected image, and then the vertical feature line of the building is extracted from the cylindrically projected image.);
extracting a plurality of feature points from each of the plurality of planar images (Paragraph 34 of Chiu teaches that the visual feature extraction module 110 comprises a convolutional neural network (CNN) trained to extract visual features from an input image stream. Additionally, FIG. 6 and paragraph 80 of Chiu teach that FIG. 6 depicts a flow diagram of a method 600 for visual localization of images of a video stream in accordance with an embodiment of the present principles. The method 600 begins at 602 during which appearance features of an image of the video stream are extracted using a first neural network, such as a deep convolutional neural network, and spatial attention.);
and obtaining the first set of feature points by projecting coordinate information associated with the plurality of feature points in each of the plurality of planar images onto the first street view image (Paragraphs 83-85 of Chiu teach that at 608, the fused features are pooled and projected into a semantic embedding space having been trained using fused appearance and semantic features of images having known locations. The method 600 can proceed to 610. At 610, a similarity measure is computed between the projected fused features and embedded, fused appearance and semantic features of images in the semantic embedding space having known locations. The method 600 can proceed to 612. At 612, a location of the image associated with the projected, fused features is predicted by determining nearest embedded, fused appearance and semantic features to the projected, fused features of the image in the semantic embedding space based on the similarity measures computed for the projected, fused features of the image.).
Regarding claim 14, Ding in view of Chen and Chiu disclose everything claimed as applied above (see claim 1), in addition, Ding in view of Chen and Chiu disclose wherein the absolute coordinate position information and direction information of the first street view image are aligned with the absolute coordinate position information of the three-dimensional model using a predefined map matching point or map matching line (Col. 10, Lines 52-60 of Ding teach that the obtaining unit is specifically configured to perform matching in a first descriptor database based on the descriptor, to obtain the localization information. The first descriptor database includes a geographical location of a first candidate point and a descriptor of the first candidate point, and the first candidate point is a candidate point whose geographical location falls within a first location range corresponding to the first localization information.).
Regarding claim 16, Ding in view of Chen and Chiu disclose everything claimed as applied above (see claim 14), in addition, Ding in view of Chen and Chiu disclose wherein the map matching line comprises a ground control line (GCL) (Col. 17, Lines 51-57 of Ding teach that in this case, a contour line of the building can be obtained, and then vertical lines in the contour line are extracted as vertical feature lines of the image. FIG. 4 is a schematic diagram of obtaining vertical feature lines of the building according to this embodiment of this application. A bold line segment in the figure indicates an obtained vertical feature line.),
and wherein each ground control line forms a corresponding pair with a line on the ground in at least one piece of three-dimensional absolute coordinate position information (Col. 7, Lines 5-13 of Ding teach that according to the visual localization method provided in this embodiment of this application, another specific storage form of the descriptor is provided. The ring array includes data corresponding to each vertical feature line and the third data indicating that no vertical feature line of the building appears.).
Regarding claim 17, Ding in view of Chen and Chiu disclose everything claimed as applied above (see claim 1), in addition, Ding in view of Chen and Chiu disclose receiving a second street view image captured at a second node within the specific area (Paragraph 64 of Chen teaches that at step 702, the mobile device generates a sub-graph from one of the plurality of second graphs. For example, a second graph generated from a street view image may be divided into several sub-graphs, such as four sub-graphs. In some embodiments, a horizontal scan is performed from left to right with a width. Additionally, paragraph 61 of Chen teaches that according to embodiments of the present disclosure, since the part of nodes 561-571 in graph 560 and the part of nodes 661-671 in graph 660 are similar, image 500 and image 600 may be determined to be matched with each other.);
generating a second visual feature map associated with the second street view image (Paragraph 66 of Chiu teaches that FIG. 4 depicts retrieval images for image searches for two cases with significant variations in viewing conditions in accordance with an embodiment of the present principles. FIG. 4, illustrates feature maps depicting the spatial attention from both modalities along with the semantic segmentation maps.);
and based on the absolute coordinate position information and direction information of the first street view image and absolute coordinate position information and direction information of the second street view image, merging the first visual feature map and the second visual feature map (Paragraph 119 of Chiu teaches that images/data captured by a plurality of sensor modalities can be communicated to an appearance detection module at which respective, specialized neural networks can be applied to determine features. The output of the appearance detection module can be communicated to a modality fusion module to fuse the appearance features and then communicated to a pooling module to pool the fused features and project the fused features into the embedding space as described above.),
wherein the absolute coordinate position information and direction information of the first street view image and the absolute coordinate position information and direction information of the second street view image are aligned with absolute coordinate position information of the three-dimensional model (Col. 16, Line 59 through Col. 17, Line 4 of Ding teach that for ease of description, the following describes a camera coordinate system and an image coordinate system. Refer to FIG. 2. The camera coordinate system is a three-dimensional rectangular coordinate system in which an optical center O of a camera is an origin and an optical axis is a z-axis. An x-axis and a y-axis of the camera coordinate system are parallel to an X-axis and a Y-axis of the image, and a z-axis of the camera coordinate system is an optical axis of the camera and is perpendicular to an image plane. An intersection point of the optical axis and the image plane is an origin of the image coordinate system, and the image coordinate system is a two-dimensional rectangular coordinate system. Additionally, Col. 21, Lines 56-67 of Ding teach that the terminal performs matching in a first descriptor database based on the descriptor, to obtain the localization information. The first descriptor database includes a geographical location of a first candidate point and a descriptor of the first candidate point, and the first candidate point is a candidate point whose geographical location falls within a first location range corresponding to the first localization information. Within the first location range determined based on the first localization information, matching is performed in a preset descriptor database based on the descriptor, so that the terminal obtains more accurate localization information within the first location range.).
Regarding claim 18, Ding in view of Chen and Chiu disclose everything claimed as applied above (see claim 1), in addition, Ding in view of Chen and Chiu disclose a non-transitory computer-readable recording medium recording instructions for executing the method (Col. 13, Lines 33-38 of Ding teach that a tenth aspect of the embodiments of this application provides a computer-readable storage medium. The computer-readable storage medium stores instructions. When the instructions are run on a computer, the computer is enabled to perform the method according to the implementations provided in the first aspect to the fourth aspect.).
Regarding claim 19, the system steps correspond to and are rejected similarly to the method steps of claim 1 (see claim 1 above). Additionally, Ding in view of Chen and Chiu disclose an information processing system (FIG. 17 and Col. 30, Lines 20-25 of Ding teach that FIG. 17 is a block diagram of a partial structure of the terminal according to this embodiment of this application. The terminal includes components such as an image capture unit 1710, a global positioning system (GPS) module 1720, a sensor 1730, a display unit 1740, an input unit 1750, a memory 1760, a processor 1770, and a power supply 1780.) comprising:
a communication module (FIG. 19 and Col. 33, Lines 54-60 of Ding teach that the receiving/sending circuit 1909 and the antenna 1910 are configured to implement a connection between the AR device and an external network. Composition units of the AR device may be coupled to each other through a communications buses, and the communications bus includes at least any one of the following: a data bus, an address bus, a control bus, an extended bus, and a local bus.);
a memory (FIG. 17 and Col. 30, Line 25 of Ding teach memory 1760);
and at least one processor connected to the memory and configured to execute at least one computer-readable program included in the memory (FIG. 17 and Col. 30, Line 25 of Ding teach a processor 1770 and Col. 32, Lines 54-60 of Ding teach that the memory 1802 may be a volatile memory or a non-volatile memory. Optionally, the processor 1801 is one or more central processing units (CPU), and the CPU may be a single-core CPU or a multi-core CPU. The processor 1801 may communicate with the memory 1802, and execute a series of instructions in the memory 1802 on the server 1800.).
Claim 2 is rejected under 35 U.S.C. 103 as being unpatentable over Ding in view of Chen and Chiu as applied to claim 1 above, and further in view of Kim et al. (Pub. No.: US 2024/0273848 A1), hereinafter Kim.
Regarding claim 2, Ding in view of Chen and Chiu disclose everything claimed as applied above (see claim 1), however, Ding in view of Chen and Chiu fail to disclose wherein the first street view image is a panoramic image generated by equirectangular projection.
Kim discloses wherein the first street view image is a panoramic image generated by equirectangular projection (Paragraph 62 teaches that the depth scanner or camera may store information about the shooting height and provide the information to the computing device 300. This shooting height information can be used to create a 3D virtual model in the computing device 300 and paragraph 63 teaches that the depth map image and the color image may be a panoramic image of a type suitable to provide a 360-degree image, for example, an equirectangular projection panoramic image. However, for convenience of explanation, hereinafter they are collectively referred to as depth map image and color image without distinction.). Since Ding in view of Chen and Chiu teach the initial method steps for obtaining street view images and generating visual feature/depth maps and Kim teaches a function for generating depth maps consisting of panoramic images and equirectangular projection panoramic images, it would have been obvious to a person having ordinary skill in the art to combine the two functions together so that any of the street view images obtained, could consist of panoramic pictures generated by an equirectangular projection if necessary.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Ding in view of Chen and Chiu to incorporate the teachings of Kim, so that the combined functions together would allow any of the street view images to also include the capabilities of utilizing panoramic images that can be generated using equirectangular projection, thus providing improved overall usability for depth map generation.
Furthermore, Ding in view of Chen, Chiu and Kim disclose and wherein the absolute coordinate position information and direction information of the first street view image are aligned with absolute coordinate position information of the three-dimensional model (Col. 16, Line 59 through Col. 17, Line 4 of Ding teach that for ease of description, the following describes a camera coordinate system and an image coordinate system. Refer to FIG. 2. The camera coordinate system is a three-dimensional rectangular coordinate system in which an optical center O of a camera is an origin and an optical axis is a z-axis. An x-axis and a y-axis of the camera coordinate system are parallel to an X-axis and a Y-axis of the image, and a z-axis of the camera coordinate system is an optical axis of the camera and is perpendicular to an image plane. An intersection point of the optical axis and the image plane is an origin of the image coordinate system, and the image coordinate system is a two-dimensional rectangular coordinate system. In addition, Col. 21, Lines 56-67 of Ding teach that the terminal performs matching in a first descriptor database based on the descriptor, to obtain the localization information. The first descriptor database includes a geographical location of a first candidate point and a descriptor of the first candidate point, and the first candidate point is a candidate point whose geographical location falls within a first location range corresponding to the first localization information. Within the first location range determined based on the first localization information, matching is performed in a preset descriptor database based on the descriptor, so that the terminal obtains more accurate localization information within the first location range.).
Claims 3, 4, 7, 8 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Ding in view of Chen and Chiu as applied to claim 1 above, and further in view of Lee (U.S. Patent: #10,546,391 B2).
Regarding claim 3, Ding in view of Chen and Chiu disclose everything claimed as applied above (see claim 1), in addition, Ding in view of Chen and Chiu disclose wherein the extracting of the first set of feature points comprises:
performing semantic segmentation on the first street view image to generate a binary mask representing a road area and a building area included in the first street view image (Col. 17, Lines 45-54 of Ding teach that optionally, first, a semantic segmentation algorithm is used to recognize and distinguish between image information of different types of objects in the image. For example, FIG. 3 is a schematic diagram of performing semantic segmentation on the image according to this embodiment of this application. The image is segmented into a building zone, a vegetation zone, a ground zone, and a sky region. In this case, a contour line of the building can be obtained, and then vertical lines in the contour line are extracted as vertical feature lines of the image. Additionally, Col. 17, Lines 57 through Col. 18, Line 2 of Ding teach that optionally, information about a regularized line segment in the image may be obtained by using a line extraction algorithm such as a line segment detector (LSD), and then a classification result mask obtained through semantic segmentation is used as a constraint to eliminate outliers inside and outside a range of the building, and retain only target segments at junctions of the building zone and the sky zone. Finally, each direct line segment to be extracted is prolongated within an effective range of a building mask in a slope direction of the line segment, to obtain a direct line segment conforming to a shape of an exterior contour of the building.). However, Ding in view of Chen and Chiu fail to disclose and extracting the first set of feature points from the first street view image using the binary mask.
Lee discloses and extracting the first set of feature points from the first street view image using the binary mask (Col. 5, Lines 4-5 teach that the feature point extraction unit 130 extracts feature points from the generated mask region.). Since Ding in view of Chen and Chiu teach the capabilities to perform semantic segmentation of a first street view image to generate a mask associated with the building areas found within the images and Lee teaches a function for using a feature point extractor to extract feature points associated with a mask region, it would have been obvious to a person having ordinary skill in the art to combine the two functions together so that a binary building mask obtained could use a feature point extraction unit to extract any feature points related to a specific mask region and/or area.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Ding in view of Chen and Chiu to incorporate the teachings of Lee, so that the combined functions together would provide a feature extraction tool to be able to focus primarily on a specific mask region and/or area, which would improve the overall target feature focus and limit the feature extractor from potentially focusing on non-important features.
Regarding claim 4, Ding in view of Chen, Chiu and Lee disclose everything claimed as applied above (see claim 3), in addition, Ding in view of Chen, Chiu and Lee disclose wherein the generating of the binary mask comprises:
converting the first street view image into a plurality of undistorted planar images (Col. 19, Lines 11-21 of Ding teach that it should be noted that, when stitching needs to be performed based on a plurality of images with overlapping information, the cylindrical projection is usually performed before stitching. Because the photographed images are all centrally projected images and the imaging surface of the image is a plane, if a plurality of images are directly stitched, a stitching result may be greatly deformed due to projection deformation. Therefore, the central projection needs to be converted to a cylindrical projection surface by performing conversion to the cylindrical projection first, so that projection surfaces of all to-be-processed images are consistent.);
and performing semantic segmentation on the plurality of undistorted planar images to detect a road area and a building area (Col. 17, Lines 45-54 of Ding teach that optionally, first, a semantic segmentation algorithm is used to recognize and distinguish between image information of different types of objects in the image. For example, FIG. 3 is a schematic diagram of performing semantic segmentation on the image according to this embodiment of this application. The image is segmented into a building zone, a vegetation zone, a ground zone, and a sky region. In this case, a contour line of the building can be obtained, and then vertical lines in the contour line are extracted as vertical feature lines of the image.).
Regarding claim 7, Ding in view of Chen, Chiu and Lee disclose everything claimed as applied above (see claim 3), in addition, Ding in view of Chen, Chiu and Lee disclose wherein the extracting of the first set of feature points from the first street view image using the binary mask comprises
extracting the first set of feature points from a partial area of the first street view image using the binary mask (Col. 4, Lines 27-42 of Lee teach that as an example of setting a mask region, a traffic line may be used as a boundary thereof. A region outside the traffic line is a region where a vehicle traveling in the same direction or a vehicle traveling on an opposite lane may be present at any time. Accordingly, when a ground surface is not shown in the image but another vehicle is shown, a problem of the ground feature points not being extracted may occur. Accordingly, the problem may be solved by limiting the mask region to a lane in which the vehicle is traveling so that an influence of a vehicle traveling in another lane is reduced. In order to set the mask region, traffic lines 550 and 560 are detected from left and right images, and a space between the vehicle and the traffic lines 550 and 560 is set as the mask region for extracting feature points.).
Regarding claim 8, Ding in view of Chen, Chiu and Lee disclose everything claimed as applied above (see claim 3), in addition, Ding in view of Chen, Chiu and Lee disclose wherein the extracting of the first set of feature points from the first street view image using the binary mask comprises:
performing filtering on a plurality of feature points extracted from the first street view image using the binary mask to extract the first set of feature points (Col. 5, Lines 6-12 of Lee teach that the feature point extraction unit 130 may detect feature points on the basis of various detection algorithms. As an example, the feature point extraction unit 130 may detect feature points using a top-hat filter, a Hough transformation algorithm, a RANSAC-based line fitting algorithm, a Kalman filter, etc. However, the present invention is not limited thereto, and various detection methods may be used.).
Regarding claim 15, Ding in view of Chen and Chiu disclose everything claimed as applied above (see claim 14), in addition, Ding in view of Chen and Chiu disclose wherein the map matching point comprises a ground control point (GCP) (Col. 24, Lines 42-46 of Ding teach that the descriptor is generated based on the vertical feature line obtained in step 902. The descriptor of the candidate point includes orientation angle information of a visible facade intersection line of a surrounding building using the candidate point as a reference point.). However, Ding in view of Chen and Chiu fail to disclose and a building control point (GCP).
Lee discloses and a building control point (GCP) (Col. 3, Lines 61-63 of Lee teaches that feature points refer to common points that may represent features of a ground surface in images that are continuously captured.). Since Ding in view of Chen and Chiu teach generating visual feature maps using coordinate position and orientation information and can match points together using a central grounded candidate point as a control reference associated with surrounding buildings and Lee teaches the functionality of applying and using numerous feature/common points which could consist of a ground point associated with a building, it would have been obvious to a person having ordinary skill in the art to combine the two features together so that any ground control point or candidate point could be used in relation to another ground feature/common point that could be associated to a point of a surrounding building.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Ding in view of Chen and Chiu to incorporate the teachings of Lee, so that the combined functions together would provide a better overall point matching function in relation to a specific ground control point/candidate point and an associated building point, which would improve spatial recognition and distance understanding between the ground point and a ground feature point associated with a building.
Furthermore, Ding in view of Chen, Chiu and Lee disclose wherein each ground control point forms a corresponding pair with a point on the ground in three-dimensional absolute coordinate position information (Col. 24, Line 63 through Col. 25, Line 9 of Ding teach that a point O is a selected candidate point on the map. The candidate point herein is manually specified map coordinate data. Because the map data is used as an input source, coordinates are known in advance. The candidate point O is used as an example. If a person stands at the point O, the point A of the building on the upper-right corner is blocked and invisible, points B, C, and D corresponding to the facade intersection lines of the building are visible, the points B, C, and D are separately connected to the point O. An intersection point between a unit circle in which the point O is located and a connection line connecting the point O and each of the points B, C, and D is a feature point of the geometric semantic descriptor. The feature point represents a vertical contour line of the building in a practical physical sense.),
and wherein each building control point forms a corresponding pair with a point on the building in three-dimensional absolute coordinate position information (Col. 3, Line 64 through Col. 4, Line 2 of Lee teach that feature points may be detected by extracting the feature points from continuous images on the basis of an optical flow and performing matching on the extracted feature points. In this case, only ground feature points may be detected by calculating a homography (a 3×3 matrix) indicating a correspondence between the ground feature points.).
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Ding in view of Chen, Chiu and Lee as applied to claim 4 above, and further in view of Yun (Pub. No.: US 2024/0273848 A1).
Regarding claim 5, Ding in view of Chen, Chiu and Lee disclose everything claimed as applied above (see claim 4), however, Ding in view of Chen, Chiu and Lee fail to disclose wherein the plurality of undistorted planar images are generated by converting the first street view image into six cube images using a perspective projection method.
Yun discloses wherein the plurality of undistorted planar images are generated by converting the first street view image into six cube images using a perspective projection method (Paragraph 24 teaches that the cube map is a composition of an environment seen from a viewpoint of a 3D object in the form of a map of six faces of a cube, and can be obtained using a panoramic image. Additionally, paragraph 59 teaches that the cube mapping unit 170 converts the finally generated panoramic image into a cube map image and applies the cube map image to the 3D vehicle model.). Since Ding in view of Chen, Chiu and Lee teach the initial method steps for generating a visual feature map associated with semantic segmentation and planar images and Yun teaches a function for converting planar panoramic images into cube map images consisting of six sides, it would have been obvious to a person having ordinary skill in the art to combine the two functions together so that any of the street view images could also be converted into six cube images for use in then being able to generate and project 360 degree views around the initial street view image.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Ding in view of Chen, Chiu and Lee to incorporate the teachings of Yun, so that the combined functions together would allow for the initial first street view image to be converted into six cube images for 360 projection use, which would help improve and eliminate any type of potential spatial or spherical distortions in relation to the fist street view image.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Strawn et al. (Pub. No.: US 2017/0356755 A1) teaches methods for generating a navigation presentation that displays navigating a route on a map.
Xu et al. (Pub. No.: US 2014/0240311 A1) teaches a method and device for transitioning between street view images which include obtaining original and target street view images and constructing a three-dimensional model corresponding to the original street view image and matching feature points between the original street view image and the target street view image.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to George Renze whose telephone number is (703)756-5811. The examiner can normally be reached Monday-Friday 9:00am - 6:00pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Xiao Wu can be reached at (571) 272-7761. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/G.R./Examiner, Art Unit 2613
/XIAO M WU/Supervisory Patent Examiner, Art Unit 2613
