DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on/after Mar. 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 USC 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3 are rejected under 35 U.S.C. 103 as being unpatentable over Porter et al. (U.S. PG-PUB 2019/0385363, 'PORTER') in view of Shteinfeld et al. (U.S. PG-PUB 2017/0302905, 'SHTEINFELD').
PNG
media_image1.png
671
519
media_image1.png
Greyscale
Regarding claim 1, PORTER discloses a method for generating, by a processor, a 3D model of a region of interest (ROI) in a geographical area, the method comprising:
([SHTEINFELD discloses this limitation.]);
classifying … the received set of planes into a set of wall planes and a set of face planes (PORTER; FIG. 4; ¶ 0044; “In step 38, system assigns face type labels to the roof detected in the retrieved image. The face type labels indicate special roof subsections that are to be handled by special rules. … the face type label can be assigned to pixels that include extensions (e.g., a carport), chimneys, crickets, terraces, etc., so that the pixels can be removed from consideration. The face type labels can also be assigned to regular roof faces and walls.”);
PNG
media_image2.png
398
570
media_image2.png
Greyscale
determining … a set of outer edges of the ROI by using intersection of the set of wall planes and the set of face planes (PORTER; ¶ 0045; “In step 40, the system assigns corner labels to the roof detected in the retrieved images. The corner labels indicate intersections between line segments. The corner labels can aid in identifying line segments that may have been missed in a first pass of identifying the line segments in the images. … the corner label can identify the missed line segments by adding constraints to the regions in which line segment intersections can happen. … the corners are assigned labels describing the type of segments that caused the corner to form. … the assigned label can identify an eave and eave corner, a flat ridge and a rake corner etc.” ¶ 0055; “FIG. 10 is an illustration of the system performing the primary line segment extraction. … there are no connections between the line segments. … a first line segment, such as an eave line segment, is not connected to a second line segment, such as another eave line segment or a rake line segment. As such, a line graph of connected segments will be used to connect the line segments. The line graph of connected segments can also be used to further refine the line segments detected in the image by using intersections between adjacent line segments to refine the endpoints, identify locations where there may be a missing line segment due to … a noisy or missing neural network output, or determine further missing line segments from newly available constraints.”);
PNG
media_image3.png
320
371
media_image3.png
Greyscale
computing … a set of individual facet geometries for each of the set of outer edges by evaluating change in shape of contour of the set of individual facet geometries at … pre-defined height(s) (PORTER; ¶ 0062; “FIG. 14 is an illustration of a valley segment that is attached to a rake segment and an eave segment being used to infer that the rake segment and eave segment should also be connected to each other. The output can be referred to an edge graph. Connecting the edges to the interior line segments can be a first pass of the line graph construction phase. However, line segments can be missing due to noise from the neural network outputs or from other sources. As such, a second pass (e.g., a secondary line extraction) can be performed. The second pass focuses on locations where our graph is incomplete. The second pass can verify whether a candidate line matches enough of a network output to be added to the graph.”); and
generating … a 3D model of the ROI by merging the computed set of individual facet geometries (PORTER; FIGS. 17-18; ¶ 0072; “In step 134, the system applies additional constraints to the line segments. The additional constraints can include rectifying parallel, perpendicular, and collinear line segments. … the 3D representation may be missing some information to reconstruct a complete valid roof. However, the system can proceed with the assumption that while the roof may be incomplete, the 3D edge sequences produced are valid. Although the assumption can be inaccurate, the assumption allows the system to proceed with the 3D reconstruction over a much more constrained search space for a final roof configuration. FIG. 18 shows an example of 3D reconstruction work flow. … the 3D reconstruction phase begins by completing any open endpoints in the line graph. Once the line graph is completed, the line graph can be used as input into a straight skeleton algorithm. The straight skeleton algorithm can infer any remaining interior roof line segments.”).
PORTER does not explicitly disclose determining … a set of planes involved in the ROI from a point cloud data using a … (RANSAC) mechanism, which SHTEINFELD discloses (SHTEINFELD; ¶ 0030-34; “… a RANSAC algorithm is used to identify planes in the point cloud. A RANSAC (random sample consensus) algorithm is an iterative method to estimate parameters of a set of observed data. A RANSAC algorithm is particularly useful if the measurement data (i.e. the depth data) comprises outliers, i.e. points of the point cloud that are distant from other points, which is usually the case for the depth data and the point cloud.”)
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to modify the method for generating a 3D model of a region of interest in a geographical area of PORTER to include the determining a set of planes involved in the ROI from a point cloud data using a (RANSAC) mechanism of SHTEINFELD. The motivation for this modification could have been to use RANSAC to find planar sets of points within a point cloud using robust estimation while ignoring a significant number of outlier points present in the overall point cloud.
Regarding claim 2, PORTER-SHTEINFELD disclose the method as claimed in claim 1, wherein the point cloud data comprises any of a LIDAR data and aerial photogrammetry data (PORTER; ¶ 0005; “This present disclosure relates to systems and methods for modeling roofs of structures using [2-D] and partial [3-D] data. The [2-D] sources can be image sources which include, but are not limited to, aerial imagery, satellite imagery, ground-based imagery, imagery taken from unmanned aerial vehicles (UAVs), mobile device imagery, etc. The [3-D] data can include … light detection and ranging (“LIDAR”), point cloud, feature triangulation, etc.”).
Regarding claim 3, PORTER-SHTEINFELD disclose the method as claimed in claim 1, wherein the processor is communicatively coupled to the [[sever]] server by a network (PORTER; ¶ 0081; “FIG. 23 is a diagram illustrating computer hardware and network components on which the system of the present disclosure could be implemented. The system can include … internal servers 224a-224n having … processor(s) and memory for executing the computer instructions and methods described above (which could be embodied as computer software 222 …). The system can also include … image storage servers 226a-226n for receiving the image data and video data. The system can also include … camera devices 228a-228n for capturing image data and video data. These systems can communicate over a communication network 230. The 3D reconstruction system 222 or engine can be stored on the internal servers 224a-224n or on an external server(s).”).
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over PORTER in view of SHTEINFELD as applied to claim 1 above, and further in view of van der Merwe et al. (U.S. PG-PUB 2013/0321392, 'MERWE').
PNG
media_image4.png
437
662
media_image4.png
Greyscale
Regarding claim 4, PORTER-SHTEINFELD disclose the method as claimed in claim 1; however, PORTER-SHTEINFELD do not explicitly disclose that the processor is … evaluating a set of inner edges, a set of peaks, and a set of valleys by using intersection of the set of wall planes and the set of face planes, which MERWE discloses (MERWE; FIG. 7; ¶ 0038; “… given a [DB] of possible roof types, these clusters may be used to match a roof type for a given building. … given a (non-exhaustive) list of roof types …, the roof analysis tool may determine that the clusters of normals for the roof in FIG. 6D may match roof type 702, which is a hip and valley type roof.” ¶ 0209; “… the roof analysis tool may perform the identification of the classification of a roof type based on each of the respective planes determined to be part of the roof of the building. … archetypal roof styles are depicted in FIG. 7 and hip and valley roof 702 [‘set of valleys’], gambrel roof 704 [‘set of inner edges’], hip roof 706, gable roof 708, mansard roof 710 [‘set of peaks’], flat roof 712, and shed roof 714.”).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to modify the method as claimed in claim 1 of PORTER-SHTEINFELD to include the evaluating a set of inner edges, a set of peaks, and a set of valleys by using intersection of the set of wall planes and the set of face planes of MERWE. The motivation for this modification could have been to use the architecturally designed seams between adjacent building façades to recognize where walls/faces intersect within a three-dimensional building model.
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over PORTER in view of SHTEINFELD as applied to claim 1 above, and further in view of Guskov et al. (U.S. PG-PUB 2015/0187130, 'GUSKOV').
PNG
media_image5.png
240
882
media_image5.png
Greyscale
Regarding claim 5, PORTER-SHTEINFELD disclose the method as claimed in claim 1; however, PORTER-SHTEINFELD do not explicitly disclose that the processor is … classifying the received set of planes based on … angle(s) of each of the set of planes with a horizontal, which GUSKOV discloses (GUSKOV; FIGS. 4A-4C; ¶ 0061-66; “… in FIG. 4A, 3D-model 400 includes a 3D building 402 and a 3D building 404. … a surface normal 406, a surface normal 408, and a surface normal 412 are surface normal vectors [‘angle(s) of each of the set of planes with a horizontal’] shown for three different polygons associated with different portions of 3D building 402. … the different types of polygons may be delineated by polygon classifier 310 based on an angle of the surface normal of each polygon relative to a reference plane … Polygons associated with a wall of 3D building 402 (or "wall polygons") may be polygons for which the surface normal is within some angle of tolerance relative to a reference plane. Conversely, polygons associated with a roof of 3D building 402 (or "roof polygons") may be polygons for which the surface normal exceeds a predetermined angle of tolerance relative to the same reference plane [‘classifying the received set of planes’]. … the reference plane is horizontal. … it may be determined by polygon classifier 310 that surface normal 406 and surface normal 408 exceed some angle of tolerance relative to the reference plane. … polygon classifier 310 can classify the polygons corresponding to surface normal 406 and surface normal 408 as roof polygons for 3D building 402. On the other hand, surface normal 412, which appears to be substantially vertical relative to the reference plane, may be determined to be within the angle of tolerance relative to the reference plane, the polygon corresponding to surface normal 112 would be classified as a wall polygon for 3D building 402. … identified roof polygons can be used to construct a connectivity graph, which in turn, can be used to identify connected roof components for a particular … roof(s) of buildings that are being represented in the 3D model. … each roof component for a particular roof may include … roof polygons that share edges. If roof polygons have discontinuous edges, the polygons are likely part of two separate roofs and would be part of separate roof meshes. … the roof components [are] a continuous mesh of polygons where each roof polygon shares … edge(s), and the perimeter of the mesh is defined as … edge(s) that are not connected to any other roof polygon. … Surface 432, indicated by shading, is a roof surface. Referring … to FIG. 4A, roof 432 [is] derived by analyzing the angles of surface normal 406 and surface normal 408 … Since the two surfaces corresponding to each surface normal vector share an edge, they … correspond to the same building. … surface 434, identified by surface normal 410, is a roof surface corresponding to building 404. FIG. 4C is a continued representation of building 404. Surface 462, indicated by shading, is a vertical surface that may overlap with a separate building … (building 402). … the surface normal of surface 462 would … be substantially horizontal relative to the reference plane and … be … within an angle of tolerance relative to the reference plane. … the polygon corresponding to surface 462 [is] classified as a wall polygon.”).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to modify the method as claimed in claim 1 of PORTER-SHTEINFELD to include the classifying the received set of planes based on … angle(s) of each of the set of planes with a horizontal of GUSKOV. The motivation for this modification could have been to use the geometric concept of a surface normal vector emanating from a plane, wherein that vector will be substantially parallel to the ground plane in the case of a vertical wall, and that vector will be substantially orthogonal to the ground plane in the case of a horizontal roof. Surface normal vectors not substantially orthogonal/parallel to the ground plane may represent pitched façades such as those present in hip/gable roofs.
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over PORTER in view of SHTEINFELD as applied to claim 1 above, and further in view of Seo et al. (U.S. PG-PUB 2024/0177421, 'SEO') and Holmes ("The DCEL Data Structure for 3D Graphics", available 2021, 'HOLMES').
Regarding claim 6, PORTER-SHTEINFELD disclose the method as claimed in claim 1; however, PORTER-SHTEINFELD do not explicitly disclose that the method of evaluating change in shape of the contour at the … pre-defined height(s), comprises the steps of:
noting and storing … event(s) in a stack, upon detection of any changes in the shape in tilt of the contour, which SEO discloses (SEO; ¶ 0094; “A costume texture is applied using a correspondence relationship between the general mesh and the UV texture. A relationship for each mesh and vertex corresponding to the UV is stored, and histories of movement information [‘noting and storing … event(s) in a stack’] about movement (change) of the mesh and vertex [‘detection of any changes in the shape in tilt of the contour’] are all stored.”).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to modify the method as claimed in claim 1 of PORTER-SHTEINFELD to include the noting and storing event(s) in a stack, upon detection of any changes in the shape in tilt of the contour of SEO. The motivation for this modification could have been to provide a recordation system for tracking the changes in a three-dimensional mesh which results from systematically connecting the points of a point cloud to form contours/mesh(es).
PORTER-SHTEINFELD-SEO do not explicitly disclose the following limitations, which are disclosed by HOLMES:
PNG
media_image6.png
325
866
media_image6.png
Greyscale
assigning, a set of faces corresponding to the noted … event(s) (HOLMES; p. 4; [See above.]);
PNG
media_image7.png
383
866
media_image7.png
Greyscale
creating, a half-edge data structure by connecting points corresponding to the noted … event(s) (HOLMES; p. 3; [See above.]); and
computing, the set of individual facet geometries by traversing the half-edge data structure in a counterclockwise [CCW] direction (HOLMES; p. 3; “The Half-Edge object contains a pointer to a Vertex, named "origin", a pointer to a Face named "face", and two pointers to Half-Edges, one named "twin" and one named "next". The origin is the vertex from which the Half-Edge starts. The face is the face on the "left" side of the Half-Edge, while the twin pointer points to the Half-Edge on the "right" side of the Half-Edge that completes its edge. The "next" pointer points to the Half-Edge that starts from h->twin->origin and ends at the next vertex in h->face, traveling counterclockwise around the boundary. This pointer allows us to traverse a polygon [‘facet geometries’], by following next pointers until we arrive back at the Half-Edge we began at.”).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to modify the method as claimed in claim 1 of PORTER-SHTEINFELD-SEO to include the assigning a set of faces corresponding to the noted event(s), the creating a half-edge data structure by connecting points corresponding to the noted event(s), and the computing the set of individual facet geometries by traversing the half-edge data structure in a CCW direction of HOLMES. The motivation for this modification could have been to implement a Doubly-Connected Edge List (DCEL), which is a data structure for efficiently storing topological information about a 2D surface (Possibly located in 3D space) (HOLMES, p. 1). Having constant time access to the neighborhood of an arbitrary point can be very useful. A normal can be computed for any given face easily, on demand, even if the vertex locations are changing. Every face touching a vertex (the "star" of the vertex) can also be traversed to easily estimate a normal for that vertex. If small portions of a mesh are locally changing, this is much easier than recomputing the normals for every vertex in the mesh. Similarly, mesh simplification for Level of Detail or compressed storage becomes easier when neighbors are easily found. Subdivision algorithms such as Loop and Catmull-Clark are relatively easy on a DCEL, especially adaptive subdivision (HOLMES, p. 7). The Examiner notes that a doubly connected edge list (DCEL) is also known as half-edge data structure, which is a data structure that represents an embedding of a planar graph in a plane.
Claims 7-8 and 10 are rejected under 35 U.S.C. 103 as being unpatentable over PORTER in view of SHTEINFELD and GUSKOV.
Regarding claim 7, PORTER-SHTEINFELD disclose a system to generate a 3D model of a region of interest, the system comprising:
a processor; and a memory storing instructions that, when executed by the processor, cause the processor to perform (PORTER; ¶ 0081; “FIG. 23 … illustrates computer hardware and network components on which the system … could be implemented. The system can include … internal servers 224a-224n having … processor(s) and memory for executing the computer instructions and methods described above (which could be embodied as computer software 222 …).”): … ([The limitations essentially repeated here are taught by PORTER and SHTEINFELD as explained above. The rationale to combine the PORTER and SHTEINFELD references will be maintained, see Office action above.])
… wherein the set of individual facet geometries are evaluated from change in shape of contour of the set of individual facet geometries at … pre-defined height(s) (PORTER; ¶ 0062; “FIG. 14 is an illustration of a valley segment that is attached to a rake segment and an eave segment being used to infer that the rake segment and eave segment should also be connected to each other. The output can be referred to an edge graph. Connecting the edges to the interior line segments can be a first pass of the line graph construction phase. However, line segments can be missing due to noise from the neural network outputs or from other sources. As such, a second pass (e.g., a secondary line extraction) can be performed. The second pass focuses on locations where our graph is incomplete. The second pass can verify whether a candidate line matches enough of a network output to be added to the graph.”);
generate, a 3D model of the ROI, wherein the computed set of individual facet geometries are merged to generate the 3D model (PORTER; FIGS. 1-2; ¶ 0034; “In step 12, the system 10 performs an imagery selection phase [which] retrieves … image(s) and metadata of the retrieved images based on a geospatial region of interest (“ROI”). In step 14, the system 10 performs a neural network inference phase [which] produces 2D outputs in pixel space, such as surface gradients, line gradients, line types, corners, etc., for … structure(s) in the retrieved image(s). In step 16, the system 10 performs a line extraction selection phase [which] processes the neural network inference outputs to create 2D line segment geometries in the pixel space. In step 18, the system 10 performs a line graph construction phase [which] processes the 2D line segment geometries along with raw 3D information to group segments into directed contour graphs of various heights [‘… facet geometries are evaluated from change in shape of contour of the set of individual facet geometries at … pre-defined height(s)’]. In step 20, the system 10 performs a 3D reconstruction phase [which] processes the output from the line graph construction phase and the metadata from the image(s) to transform the line data into 3D line segment geometries in world space [‘generate, a 3D model of the ROI, wherein the computed set of individual facet geometries are merged to generate the 3D model’]. … FIG. 2 illustrates the method 10.”); and
PORTER-SHTEINFELD do not explicitly disclose displaying, the generated 3D model on a computing device, which GUSKOV discloses (GUSKOV; ¶ 0034; “… images retrieved and presented by image viewer 120 are 3D representations of various real-world objects associated with a geographical location. … 3D representations of buildings from a city block may be generated based on images of a major city taken by satellite at various angles. … images retrieved and presented by image viewer 120 are 3D graphical models that can be presented on the client display.”).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the system to generate a 3D model of a region of interest of PORTER-SHTEINFELD-GUSKOV to include the displaying the generated 3D model on a computing device of GUSKOV. The motivation for this modification could have been to help a human user visually perceive and manipulate a three-dimensional model using an intuitive graphical user interface.
Regarding claim 8, PORTER-SHTEINFELD-GUSKOV disclose the system as claimed in claim 7, wherein the point cloud data comprises any of a LIDAR data and aerial photogrammetry data (PORTER; ¶ 0005).
Regarding claim 10, PORTER-SHTEINFELD-GUSKOV disclose the system as claimed in claim 7, wherein the processor is … classifying the received set of planes based on … angle(s) of each of the set of planes with a horizontal (GUSKOV; FIGS. 4A-4C; ¶ 0061-66).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to modify the system as claimed in claim 7 of PORTER-SHTEINFELD-GUSKOV to include the classifying the received set of planes based on … angle(s) of each of the set of planes with a horizontal of GUSKOV. The motivation for this modification could have been to use the geometric concept of a surface normal vector emanating from a plane, wherein that vector is substantially parallel to the ground plane in the case of a vertical wall, and that vector is substantially orthogonal to the ground plane in the case of a horizontal roof. Surface normal vectors not substantially orthogonal/parallel to the ground plane may represent pitched façades such as those present in hip/gable roofs.
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over PORTER in view of SHTEINFELD and GUSKOV as applied to claim 7 above, and further in view of MERWE.
Regarding claim 9, PORTER-SHTEINFELD-GUSKOV disclose the system as claimed in claim 7; however, PORTER-SHTEINFELD-GUSKOV do not explicitly disclose that the processor is … evaluating a set of inner edges, a set of peaks, and a set of valleys by using intersection of the set of wall planes and the set of face planes, which MERWE discloses (MERWE; FIG. 7; ¶ 0038; ¶ 0209).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to modify the system as claimed in claim 7 of PORTER-SHTEINFELD-GUSKOV to include the evaluating a set of inner edges, a set of peaks, and a set of valleys by using intersection of the set of wall planes and the set of face planes of MERWE. The motivation for this modification could have been to use the architecturally designed seams between adjacent building façades to recognize where walls/faces intersect within a three-dimensional building model.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JONATHAN M COFINO whose telephone number is (303) 297-4268. The examiner can normally be reached Monday-Friday 10A-4P MT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kent Chang can be reached at 571-272-7667. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JONATHAN M COFINO/ Examiner, Art Unit 2614
/KENT W CHANG/ Supervisory Patent Examiner, Art Unit 2614