DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 11/10/2025 has been entered.
Claim Status:
Applicant’s amendment and arguments filed on 11/10/2025 have been fully considered.
Claims 1, 4-8, 10-15 and 18-20 are pending.
Claims 1 and 15 are amended.
Claims 2-3, 9 and 16-17 are cancelled.
No claim is new.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 4, 8, 10-12, 15 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Adan et al. (“Reconstruction of Wall Surfaces Under Occlusion and Clutter in 3D Indoor Environments” CMU-RI-TR-10-12 Robotics Institute, Carnegie Mellon University Pittsburgh, Pennsylvania 15213, April 2010, “Adan” ) in view of Jun (“A piecewise hole filling algorithm in reverse engineering” Department of Mechanical Engineering, Sejong University, Gunja-dong 98, Gwangjin-gu, Seoul 143-747, South Korea Received 30 December 2003; received in revised form 16 June 2004; accepted 17 June 2004, “Jun”) and Criminisi et al. ( "Object removal by exemplar-based inpainting." 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings.. Vol. 2. IEEE, 2003 “Criminisi”).
Regarding Claim 1 Adan teaches A system (Page 24 Pentium III) comprising:
At least one processor; and at least one memory storing components executable by the at least one processor, the components including: (Page 24, Pentium III computer has memory and processor
PNG
media_image1.png
143
1393
media_image1.png
Greyscale
)
A first identification component (integral part of processor of Pentium III) that evaluates captured three-dimensional (3D) data associated with an indoor environment and identifies flat surfaces included in the captured 3D data (Page 3 1. Introduction 3rd paragraph “The content of this document is framed in the BIM context. The first goal of this work is to automatically identify essential parts of a building by using range data under occlusion circumstances. Essential parts are walls, ceiling and floors”);
Even though Adan talks about 3d mesh data by referring Perez reference (Page 22 r 6. Filling Occluded Parts of the Wall The gap filling technique that we use here was published in [Perez 2008]. The technique published in [Perez 2008] was originally applied to the filling of gaps in 3D meshes”) but is silent about details of 3d mesh data;
However Jun teaches hole filing algorithm with 3d mesh data including multiple polygons (ABSTRACT “Missing scanned data cause holes in the created triangular mesh, so that a hole-free mesh model is prerequisite for fitting watertight surfaces”…. See Introduction 4th paragraph :”The contribution of this paper is to build an automated algorithm that fills complex holes. It identifies polygonal holes in a 3D triangular mesh and consecutively fills them regardless of the shape complexity of the holes.”);
Jun and Adan are analogous art as both of them are related to hole filling.
Therefore it would have been obvious for an ordinary skilled person in the art before the effective filing date of claimed invention to have modified Adan by using mesh data including multiple polygons as captured 3d data for hole filing as explicitly taught by Jun.
The motivation for the above is to enhance the applicability of Adan as meshes are faster and efficient in editing.
Adan modified by Jun teaches a second identification component (integral part of processor of Pentium III of Adan) that evaluates the 3D mesh data and identifies an object included in the 3D mesh data, the object being connected to at least one flat surface of the flat surfaces in the 3D mesh data (Adan Page 3 4th paragraph 1. Introduction “The second objective of this work has two aims: firstly, to recognize which parts of the occluded zones belong to the wall and which do not”; Page 17 1st paragraph 5. Segmentation of Occluded Regions “Basically, the most important holes in a wall are doors, windows, closets and built-in shelves. In this work, we tackle the problem of recognizing this type of object”);
However Adan modified by Jun is silent about removes the object and at least a portion of the at least one flat surface from the 3D mesh data based on boundaries in the 3D mesh data to cause the at least one flat surface in the 3D mesh data to have omitted 3D mesh data.
Criminisi teaches removes object and at least a portion of at least one flat surface based on boundaries in data to cause the at least one flat surface to have omitted data (Abstract : “A new algorithm is proposed for removing large objects from digital images. The challenge is to fill in the hole that is left behind in a visually plausible way”. Figure 7 “. (a) Original image. (b) The target region has been selected and marked in red. (c) Results of filling by concentric layers. (d) Results of filling with our algorithm”. In fig. 7 signboard and part of the handle is removed and fills the region).
Criminisi and Adan modified by Jun are analogous art as both of them are related to hole filling.
Therefore it would have been obvious for an ordinary skilled person in the art before the effective filing date of claimed invention to have modified Adan modified by Jun by removing the object and at least a portion of the at least one flat surface from the 3D mesh data based on boundaries in the 3D mesh data to cause the at least one flat surface in the 3D mesh data to have omitted 3D mesh data similar to removing object and at least a portion of at least one flat surface based on boundaries in data to cause the at least one flat surface to have omitted data as taught by Criminisi.
The motivation for the above is to reconstruct an environment without hazard.
Adan modified by Jun and Criminisi teaches a data generation component (Integral part of processor of Pentium III of Adan) that identifies the omitted 3D mesh data of the at least one flat surface based on analyzing boundary edges between the removed object and the at least one flat surface and generates additional 3D mesh data for the omitted 3D mesh data of the at least one flat surface by extending the at least one flat surface across the omitted 3D mesh data using a surface parameterization of the at least one flat surface ( Adan identifies the omitted 3D mesh data of the at least one flat surface based on analyzing boundary edges of holes of the at least one flat surface See Adan ABSTRACT:
PNG
media_image2.png
69
1034
media_image2.png
Greyscale
PNG
media_image3.png
267
1026
media_image3.png
Greyscale
Adan modified by Jun and Criminisi creates a hole by removing an object from the surface. So the hole boundary edges are the boundary edges between the removed object and the at least one flat surface.
June generates additional 3D mesh data for the omitted 3D mesh data of the at least one flat surface by extending the at least one flat surface across the omitted 3D mesh data using a surface parameterization of the at least one flat surface. See June Page 266 left column: “(i) apply planar triangulation algorithm to each subhole. (ii) apply computed intersection parameters to the original edges (iii) sub-division † until no sub-hole leaves unfilled † update topology information 3.1. Partitioning a complex hole into several sub-holes The intersection points from the projected boundary edges are computed as follows (Fig. 8). Let one edge represented by parametric form….. The parameters u and v are increased from 0 to 1. If r0, r1, and p0, p1 are the end points of two boundary edges, respectively, rA, rB, and pA, pB are derived by rA=r0, pA=p0, PB= p1-p0.” Here the edges are parametric that means each edge of has parameter of the flat surface.
Jun Page 268 right Column 4. Examples “Fig. 13a shows the triangular mesh with shaded image. Fig. 13b and c shows the enlarged image of complex holes and their boundary edges, respectively. The reconstructed final model after applying the proposed hole filling algorithm is shown in Fig. 13d–f. Fig. 14 shows the results of closing a model human having 85,578 vertices and 170,795 triangles. Fig. 14a and b show missing data in the reconstructed triangular polygon and their hole boundary lines, respectively. The model has 12 simple holes and three complex holes. The reconstructed model after hole triangulation is shown in Fig. 14c”).
Claim 15 is directed to a method and its steps are similar in scope and functions of the system claim 1 and claim 15 is also rejected with the same rationale as specified in the rejection of claim 1.
Regarding claim 4 Adan modified by Jun and Criminisi teaches wherein the data generation component identifies the object based on proximity data in relation to the at least one flat surface (Jun Page 265 Left column 2.2. Hole detection “The first step of hole filling process is to identify holes in the triangular mesh. Since the implemented data structure is based upon the triangle-based topology that stores three neighbouring triangles (Fig. 2a), all boundary edges can be automatically identified by checking all the neighbouring triangles”).
Regarding claim 8 Adan modified by Jun and Criminisi teaches wherein the first identification component identifies a flat surface associated with the object (Adan
PNG
media_image4.png
119
1092
media_image4.png
Greyscale
).
Regarding claim 10 Adan modified by Jun and Criminisi teaches wherein the first identification component classifies the flat surface as a floor, a wall or a ceiling based on orientation information associated with the 3D mesh data (Adan Page 3, 1. Introduction :
PNG
media_image5.png
114
1073
media_image5.png
Greyscale
).
Regarding claim 11 Adan modified by Jun and Criminisi teaches wherein the second identification component defines a boundary in the mesh data associated with the object (Jun Page 265 Left column 2.2. Hole detection “The first step of hole filling process is to identify holes in the triangular mesh. Since the implemented data structure is based upon the triangle-based topology that stores three neighbouring triangles (Fig. 2a), all boundary edges can be automatically identified by checking all the neighbouring triangles”).
Regarding claim 12 Adan modified by Jun and Criminisi teaches a modification component (Integral part of Adan’s processor) that modifies geometry data or texture data for the at least one flat surface (Jun page 268 3.3. Update of the created vertices and triangles “After subdividing the holes, the newly created vertices and triangles need to be added to their respective lists”).
Regarding claim 20 Adan modified by Jun and Criminisi teaches further comprising identifying a rectangular opening associated with the at least one flat surface (Adan
PNG
media_image6.png
85
888
media_image6.png
Greyscale
).
Claims 5 and 18-19 are rejected under 35 U.S.C. 103 as being unpatentable over Adan modified by Jun and Criminisi as applied to claim 1 above, and further in view of Valero et al. (“Automatic Construction of 3D Basic-Semantic Models of Inhabited Interiors Using Laser Scanners and RFID Sensors” Sensors 2012, 12, 5705-5724; doi:10.3390/s120505705, May 3 2012, “Valero”).
Regarding claim 5 Adan modified by Jun and Criminisi is silent about wherein the second identification component generates an object relation identifier for the object that defines a spatial relationship between the object and the at least one flat surface.
Valero teaches generating an object relation identifier for an object that defines a spatial relationship between the object and at least one flat surface (Page 5713 shows a table where it shows type of element for opening as well as for other objects
PNG
media_image7.png
457
1085
media_image7.png
Greyscale
);
Valero and Adan modified by Jun and Criminisi are analogous art as both of them are related to 3d model processing.
Therefore it would have been obvious for an ordinary skilled person in the art before the effective filing date of claimed invention to have modified Adan modified by Jun and Criminisi by having Adan’s second identification component (integral part of processor Pentium III) to generate an object relation identifier for an object that defines a spatial relationship between the object and at least one flat surface as taught by Valero.
The motivation for the above is to provide the users capability of interactive modification of images by selecting the object relation identifier.
Regarding claim 18 Adan modified by Jun and Criminisi teaches wherein the identifying the object included in the 3D mesh data as shown above but is silent about identifying the object included in the 3D mesh data based on texture data associated with the object and the at least one flat surface.
Valero teaches identifying object based on texture data associated with the object and the at least one flat surface (Page 5711
PNG
media_image8.png
74
1159
media_image8.png
Greyscale
PNG
media_image9.png
114
1169
media_image9.png
Greyscale
Here recognition of component is happening based on data which is geometry and texture).
Valero and Adan modified by Jun are analogous art as both of them are related to 3d model processing.
Therefore it would have been obvious for an ordinary skilled person in the art before the effective filing date of claimed invention to have modified Adan modified by Jun by identifying object included in the 3D mesh data based on texture data associated with the object and the at least one flat surface similar to identifying object based on texture data associated with the object and the at least one flat surface as taught by Valero.
The motivation for the above is to provide the users the capability of interactive modification of images by selecting the object identifier.
Regarding claim 19 Adan modified by Jun and Criminisi is silent about generating object relation data associated with the object and the at least one flat surface.
Valero teaches generating object relation data associated with the object and the at least one flat surface (Page 5713
PNG
media_image10.png
411
975
media_image10.png
Greyscale
Valero generates object relation data);
Valero and Adan modified by Jun and Criminisi are analogous art as both of them are related to 3d model processing.
Therefore it would have been obvious for an ordinary skilled person in the art before the effective filing date of claimed invention to have modified Adan modified by Jun and Criminisi by generating object relation data associated with the object and the at least one flat surface as taught by Valero.
The motivation for the above is to provide the users the capability of modification of portions based on the data of neighboring portions.
Claims 6-7 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Adan modified by Jun and Criminisi as applied to claim 1 above, and further in view of Colburn et al. (“Image-Based Remodeling”, IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS. VOL 19, NO. 1, JANUARY 2013, “Colburn”).
Regarding claim 6 Adan modified by Jun and Criminisi is silent about a modification component that modifies position or orientation of the object.
Colburn teaches modifying position or orientation of object (Fig. 3 a-e shows modifying position/orientation of portion of the 3d data
PNG
media_image11.png
516
888
media_image11.png
Greyscale
Page 64
PNG
media_image12.png
224
721
media_image12.png
Greyscale
);
Colburn and Adan modified by Jun and Criminisi are analogous art as both of them are related to 3d model processing.
Therefore it would have been obvious for an ordinary skilled person in the art before the effective filing date of claimed invention to have modified Adan modified by Jun and Criminisi by having Adan’s modification component (integral part of processor Pentium III) to modify position or orientation of object as taught by Colburn.
The motivation for the above is to make sure that the other portion of 3d data nicely fits with the portion of the 3d data of a surface.
Regarding claim 7 Adan modified by Jun, Criminisi and Colburn teaches wherein the modification component modifies the position or the orientation of the object based on input received via a user interface(Colburn Page 64
PNG
media_image12.png
224
721
media_image12.png
Greyscale
).
Regarding claim 13 Even though Adan modified by Jun and Criminisi teaches wherein the modification component modifies the geometry data or the texture data for the at least one flat surface ((Jun page 268 3.3. Update of the created vertices and triangles “After subdividing the holes, the newly created vertices and triangles need to be added to their respective lists”) but is silent about based on input data received via a user interface.
Colburn teaches modifies the geometry data or the texture data for the at least one flat surface based on input data received via a user interface (Page 56 Right Column
PNG
media_image13.png
85
709
media_image13.png
Greyscale
PNG
media_image14.png
303
975
media_image14.png
Greyscale
);
Colburn and Adan modified by Jun and Criminisi are analogous art as both of them are related to 3d model processing.
Therefore it would have been obvious for an ordinary skilled person in the art before the effective filing date of claimed invention to have modified Adan modified by Jun and Criminisi by modifying the geometry data or the texture data for the at least one flat surface based on input data received via a user interface as taught by Colburn.
The motivation for the above is to provide users interactive control in modification of the texture or geometry data of the window or door opening.
Claim 14 is rejected under 35 U.S.C. 103 as being unpatentable over Adan modified by Jun and Criminisi as applied to claim 12 above, and further in view of Ishiyama et al. ( US patent Publication: 20090129665, “Ishiyama”).
Regarding claim 14, Adan as modified by Jun and Criminisi silent about, wherein the modification component determines illumination data associated with the texture data that is applied when modifying the texture data.
However, Ishiyama teaches, a modification component determines illumination data associated with a texture data that is applied when modifying the texture data. ([0039]….“a first illumination basis calculation means (realized by, e.g., an illumination basis calculation means 201) for calculating an individual illumination basis representing a variation in the luminance value of each portion of the 3-dimensional surface of the individual object under various illumination conditions based on the individual 3-dimensional shape data and individual texture data generated by the individual data generation means; a reproduced image generation means (realized by, e.g., an illumination correction means 203) for generating a reproduced image which is an image reproducing the same illumination condition as an input image using the individual illumination basis group calculated by the first illumination basis calculation means; and a parameter update means (realized by, e.g., a parameter update means 204) for repeatedly updating the 3-dimensional shape parameter and texture parameter to be given to the generalized 3-dimensional object model such that the reproduced image generated by the reproduced image generation means is close to the input image until a predetermined convergence condition (for example, until the update amount of each parameter becomes smaller than a predetermined threshold value) is satisfied.”)
Adan as modified by Jun and Criminisi and Ishiyama are analogous as they are from the field of image processing.
Therefore it would have been obvious for an ordinary skilled person in the art before the effective filing date of the claimed invention to have modified Adan as modified by Jun and Criminisi to have the modification component determine illumination data associated with the texture data that is applied when modifying the texture data as taught by Ishiyama.
The motivation to include the modification is to make sure that the modified image have the similar illumination of the original image.
Response to Arguments
Applicant’s arguments, see remarks, Pages 6-8, filed 11/10/2025, with respect to the rejection(s) of claim(s) 1 and 15 under 35 USC 103 have been fully considered and are not persuasive. Therefore the rejection has been maintained. Applicant argues, see remarks Pages 7-8 “The prior art combination fails to teach this supported approach. Adan teaches filling occluded regions behind objects but does not teach removing objects or portions of surfaces first, then identifying omitted mesh data based on boundary analysis between removed objects and surfaces. Jun teaches filling holes that already exist in triangular meshes from incomplete scanning data, not the claimed approach of generating mesh data for omitted areas created by deliberate removal of objects and surface portions using surface parameterization. Criminisi teaches removing objects from 2D digital images using exemplar-based inpainting techniques, but operates entirely in the 2D image domain and does not teach generating 3D mesh data or extending flat surfaces using surface parameterizations. The combination provides no teaching of a data generation component that identifies the omitted 3D mesh data of the at least one flat surface based on analyzing boundary edges between the removed object and the at least one flat surface, and generates additional 3D mesh data by extending the at least one flat surface across the omitted 3D mesh data using a surface parameterization of the at least one flat surface, as recited by claims 1 and 15.”
Examiner replies, removing objects from the flat surface is integrated from Criminisi See Criminisi, Abstract : “A new algorithm is proposed for removing large objects from digital images. The challenge is to fill in the hole that is left behind in a visually plausible way”. Figure 7 “. (a) Original image. (b) The target region has been selected and marked in red. (c) Results of filling by concentric layers. (d) Results of filling with our algorithm”. In fig. 7 signboard and part of the handle is removed and fills the region)
After integrating the removal of an object from Ada’s surface, Ada’s surface has a hole. Adan modified by Jun and Criminisi creates a hole by removing an object from the surface. So the hole boundary edges are the boundary edges between the removed object and the at least one flat surface.
Now June generates additional 3D mesh data for the omitted 3D mesh data of the at least one flat surface by extending the at least one flat surface across the omitted 3D mesh data using a surface parameterization of the at least one flat surface. See June Page 266 left column: “(i) apply planar triangulation algorithm to each subhole. (ii) apply computed intersection parameters to the original edges (iii) sub-division † until no sub-hole leaves unfilled † update topology information 3.1. Partitioning a complex hole into several sub-holes The intersection points from the projected boundary edges are computed as follows (Fig. 8). Let one edge represented by parametric form….. The parameters u and v are increased from 0 to 1. If r0, r1, and p0, p1 are the end points of two boundary edges, respectively, rA, rB, and pA, pB are derived by rA=r0, pA=p0, PB= p1-p0.” Here the edges are parametric that means each edge has parameter of the flat surface.
Finally June Jun Page 268 right Column 4 creates final reconstructed mesh using the additional 3d mesh data created by extending the surface across the omitted 3D mesh data using a surface parameterization of the at least one flat surface. See Jun Page 268 right Column 4. Examples “Fig. 13a shows the triangular mesh with shaded image. Fig. 13b and c shows the enlarged image of complex holes and their boundary edges, respectively. The reconstructed final model after applying the proposed hole filling algorithm is shown in Fig. 13d–f. Fig. 14 shows the results of closing a model human having 85,578 vertices and 170,795 triangles. Fig. 14a and b show missing data in the reconstructed triangular polygon and their hole boundary lines, respectively. The model has 12 simple holes and three complex holes. The reconstructed model after hole triangulation is shown in Fig. 14c”.
Adan teaches, a data generation component (Integral part of processor of Pentium III of Adan) that identifies the omitted 3D mesh data of the at least one flat surface based on analyzing boundary edges of holes of the at least one flat surface See Adan ABSTRACT:
PNG
media_image2.png
69
1034
media_image2.png
Greyscale
PNG
media_image3.png
267
1026
media_image3.png
Greyscale
The teaching of removing an object from the flay surface is integrated from Criminisi and June’s teaching generating additional 3D mesh data for the omitted 3D mesh data of the at least one flat surface by extending the at least one flat surface across the omitted 3D mesh data using a surface parameterization of the at least one flat surface is applied in the combination to have the Adam as modified by June and Criminisi to teach a data generation component that identifies the omitted 3D mesh data of the at least one flat surface based on analyzing boundary edges between the removed object and the at least one flat surface, and generates additional 3D mesh data by extending the at least one flat surface across the omitted 3D mesh data using a surface parameterization of the at least one flat surface.
In response to applicant’s arguments for the dependent claims, see remarks Pages 8-11, examiner refers applicant to the reply given for independent claim above as applicant has not provided any additional argument for the dependent claims.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SAPTARSHI MAZUMDER whose telephone number is (571)270-3454. The examiner can normally be reached 8 am-4 pm PST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Said Broome can be reached at (571)272-2931. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SAPTARSHI MAZUMDER/Primary Examiner, Art Unit 2612