CTNF 18/866,542 CTNF 100902 DETAILED ACTION Notice of Pre-AIA or AIA Status 07-03-aia AIA 15-10-aia The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA. Claim Objections 07-29-01 AIA Claim 9 is objected to because of the following informalities: Claim 9 Line 3 reciting, “the interiror of the structure” should read “the interior of the structure” . Appropriate correction is required. Claim Rejections - 35 USC § 101 Claim 7 is rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. The claim does not fall within at least one of the four categories of patent eligible subject matter because claim 7 recites, “A program causing a computer to function as the three-dimensional reconstruction device according to claim 1.” Computer programs claimed as computer listings per se, ie., the descriptions or expressions of the programs, are not physical “things.” They are neither computer components nor statutory processes, as they are not “acts” being performed. Such claimed computer programs do not define any structural and functional interrelationships between the computer program and other claimed elements of a computer which permit the computer program’s functionality to be realized. In contrast, a claimed non-transitory computer-readable medium encoded with a computer program is a computer element which defines structural and functional interrelationships between the computer program and the rest of the computer which permit the computer program’s functionality to be realized, and is thus statutory. Accordingly, it is important to distinguish claims that define descriptive material per se from claims that define statutory inventions. Claim Rejections - 35 USC § 103 07-20-aia AIA The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action: A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made. 07-21-aia AIA Claim 1, 4, and 7-9 are r ejected under 35 U.S.C. 103 as being unpatentable over H utchcroft et al. (US 12045951 B2) (Hereinafter referred to as Hutchcroft) in view of Yang et al. (“DuLa-Net: A Dual-Projection Network for Estimating Room Layouts from a Single RGB Panorama”) (Hereinafter referred to as Yang) and in further view of Lorenzo (US 20190327413 A1). R egarding Claim 1, Hutchcroft discloses A three-dimensional reconstruction device comprising a processor configured to execute operations comprising: (See Col 2 Lines 14 – 15, “The present disclosure describes techniques for using computing devices to perform automated operations . . .” Also see Col 34 Lines 43 – 47, “In the illustrated embodiment, an embodiment of the IIMIGM system 340 executes in memory 330 in order to perform at least some of the described techniques, such as by using the processor(s) 305 to execute software instructions of the system 340. . .”) acquiring a panoramic image obtained by imaging an interior of a structure as a target structure; (See Col 10 Lines 21 – 28, “FIG. 1A further illustrates an IIMIGM (Inter-Image Mapping Information Generation Manager) system 140 that is executing on one or more server computing systems 180 to analyze visual data of acquired target images (e.g., panorama images 165 ) acquired in each of some or all building rooms or other building areas , and to use results of the analysis to further generate and provide building floor plans 145 and/or other mapping-related information . . .” Also see Fig. 2F stating “ multiple panorama images 241 for buildings(s) , optionally with floor / ceiling height information . . .”) acquiring a ceiling height of the structure; (See Fig. 2F stating “multiple panorama images 241 for buildings(s), optionally with floor / ceiling height information . . .”) generating a plan view of the interior of the structure based on the panoramic image and the ceiling height; (See Col 10 Lines 21 – 28, “FIG. 1A further illustrates an IIMIGM (Inter-Image Mapping Information Generation Manager) system 140 that is executing on one or more server computing systems 180 to analyze visual data of acquired target images (e.g., panorama images 165) acquired in each of some or all building rooms or other building areas, and to use results of the analysis to further generate and provide building floor plans 145 and/or other mapping-related information . . .” Also see Col 18 Lines 11 – 18, “ A building floor plan with associated room shape information may have various forms in various embodiments, such as a 2D (two-dimensional) floor map of the building (e.g., an orthographic top view or other overhead view of a schematic floor map that does not include or display height information) and/or a 3D (three-dimensional) or 2.5D (two and a half-dimensional) floor map model of the building that does display height information .” In this case, generating floor plans correspond to “generating a plan view”, and this would be based on the panorama image. This generation can also be considered to be based on the “ceiling height” as it can display height information.) determining coordinate values of the panorama images ; and (See Col 5 Lines 12 – 20, “determine global alignments of those acquisition locations (e.g., determining the acquisition locations of those some or all target images in a global common coordinate system , whether in a relative or absolute manner), and using the images' globally aligned acquisition locations and associated structural layout information to form a 2D and/or 3D floor plan (whether partial or complete, such as based on which target images are acquired and/or included in the common coordinate system ).”) performing a three-dimensional reconstruction of the structure based on the determined coordinate values of the panorama images . (See Col 5 Lines 32 – 35, “ . . . in other embodiments the determined structural layout information may include a partial or complete 3D structure for visible room(s) or other building area(s) . . .” Also see Col 5 Lines 47 - 53, “In addition, in embodiments in which such room shapes are generated, they may be further used as part of one or more additional operations, such as when generating a floor plan (e.g., to generate a 3D model floor plan using 3D room shapes , to generate a 2D floor plan by fitting 3D room shapes together and then removing height information, etc.) . . .” See Col 18 Lines 23 – 34, “For example, in at least some embodiments, an Inter-Image Mapping Information Generation Manager (IIMIGM) system may analyze various target images acquired in and around a building in order to automatically determine room shapes of the building's rooms (e.g., 3D room shapes , 2D room shapes, etc., such as to reflect the geometry of the surrounding structural elements of the building)—the analysis may include, for example, automated operations to ‘register’ the camera positions for the images in a common frame of refence so as to ‘align’ the images and to estimate 3D locations and shapes of objects in the room . . .” Lastly, see Col 5 Lines 12 – 20 teaching to put the target images (panorama images) into a common coordinate system. In this case, generating a 3D model floor plan using 3D room shapes corresponds to “a three-dimensional reconstruction of the structure”. Additionally, Hutchcroft teaches to register the camera positions of the images in a common frame of reference in order to estimate 3D locations and shapes of the objects. This can be considered as the three-dimensional reconstruction being based on “determined coordinate values”.) However, Hutchcroft fails to explicitly disclose acquiring a drawing of the interior of the structure and a ceiling height of the structure; performing alignment between the generated plan view and the drawing ; determining coordinate values of the plan view ; and performing a three-dimensional reconstruction of the structure based on the determined coordinate values of the plan view . Yang teaches acquiring a drawing of the interior of the structure and a ceiling height of the structure; (See Page 2 Right Column Paragraph 1, “In terms of room layout assumptions, a popular choice is the ”Manhattan world” assumption [4], meaning that all walls are aligned with a global coordinate system [4, 23].” Also see Page 3 Fig. 2 stating, “Then, our system estimates a 2D floor plan by fitting a Manhattan-world aligned polygon to a weighted average of the three floor plans, which is further extruded using the predicted layout height to obtain the final 3D room layout.” In this case, the Manhattan-world aligned polygon would correspond to “a drawing of the interior of the structure” as it is considered as being representative of a room layout. Note that Yang also includes acquiring a predicated layout height which corresponds to “a ceiling height of the structure”.) performing alignment between the generated plan view and the drawing; (See Page 3 Fig. 2 stating, “Then, our system estimates a 2D floor plan by fitting a Manhattan-world aligned polygon to a weighted average of the three floor plans , which is further extruded using the predicted layout height to obtain the final 3D room layout.” In this case, fitting a Manhattan-world aligned polygon to the floor plan would correspond to “performing alignment between the generated plan view and the drawing”.) It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Hutchcroft with Yang to include acquiring a Manhattan-world aligned polygon representative of the room layout and fitting the generated floor plan to that polygon. The motivation to combine Hutchcroft with Yang would have been obvious as both arts are within the same field of using panorama images to generate floor plans (See Yang Abstract). The benefit of fitting the floor plan to the Manhattan-world aligned polygon can be seen in Yang Page 5 Fig. 3 in which a polygon shape is fit onto the floor plan region, resulting in a cleaner and more standard final floor plan. However, Hutchcroft in view of Yang still fail to explicitly disclose determining coordinate values of the plan view ; and performing a three-dimensional reconstruction of the structure based on the determined coordinate values of the plan view . Lorenzo teaches determining coordinate values of the plan view; and (See [0041], “ Any specific location associated with the floor plan 300 (i.e. a camera viewing location) has both an X and a Y coordinate . The X coordinate corresponds to a left-right position on the viewed page, and the Y coordinate corresponds to a top-bottom position on the viewed page. Thus, and X-Y coordinate pair specifies a specific position on the viewed page, or a specific position on the 2D floor plan 300 .”) performing a three-dimensional reconstruction of the structure based on the determined coordinate values of the plan view. (See [0041] teaching 2D X-Y coordinates for the floor plan. See [0078], “Once the 2D coordinates and orientations have been determined 732, the X-Y coordinates are converted into 3D model coordinates 736.” Lastly, see [0043], “At least two common points are specified in both the 2D floor plan 300 and a 3D model 400, in order to align the 2D floor plan 300 with the 3D model 400 . The common points are points in space that represent the exact same position in both the 2D floor plan 300 and the 3D model 400.” Here, Lorenzo teaches that the 2D coordinates (coordinate values) of the floor plan (plan view) can be used in aligning the floor plan and 3D model. Note that Lorenzo does not explicitly teach the generation of a 3D model. It should be noted however, that Hutchcroft does teach 3D reconstruction. See Hutchcroft Col 5 Lines 47 – 53. Thus, by combining the two arts, when considering that the alignment between the 2D floor plan and 3D model taught by Lorenzo is apart of the 3D reconstruction process, then the combination teaches “performing a three-dimensional reconstruction of the structure based on the determined coordinate values of the plan view.”) It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Hutchcroft in view of Yang with Lorenzo to include determining coordinate values for the plan view and using that in the 3D reconstruction of the structure. The motivation to combine Hutchcroft in view of Yang with Lorenzo would have been obvious as all three arts are within the same field of capturing panoramic images (See Lorenzo [0002]) and processing floor plans (See Lorenzo [0009]). The benefit of having coordinate values for the floor plans and using that in the 3D reconstruction of the structure is that it results in the 3D reconstruction being more accurate and geometrically aligned, enabling proper spatial relationships in the 3D model. In particular, one can establish coordinate systems between the 2D floor plan and 3D model, and have scaling values can be defined (See Lorenzo [0048]). Regarding Claim 4, Hutchcroft in view of Yang and Lorenzo disclose A method for reconstructing a three-dimensional reconstruction executed by a three-dimensional reconstruction device, comprising: (See Hutchcroft Claim 1, “A computer-implemented method comprising: . . .” Also see Hutchcroft Col 34 Lines 43 – 50, “In the illustrated embodiment, an embodiment of the IIMIGM system 340 executes in memory 330 in order to perform at least some of the described techniques, such as by using the processor(s) 305 to execute software instructions of the system 340 in a manner that configures the processor(s) 305 and computing system(s) 300 to perform automated operations that implement those described techniques.”) acquiring a panoramic image obtained by imaging an interior of a target structure; acquiring a drawing of the interior of the structure and a ceiling height of the structure; generating a plan view of the interior of the structure based on the panoramic image and the ceiling height; performing alignment between the generated plan view and the drawing and determining coordinate values of the plan view; and performing three-dimensional reconstruction of the structure based on the determined coordinate values of the plan view. (The above limitations are similar to those of Claim 1 and is therefore rejected under a similar rationale as that of Claim 1.) Regarding Claim 7, Hutchcroft in view of Yang and Lorenzo disclose A program causing a computer to function as the three-dimensional reconstruction device according to claim 1. (See Hutchcroft Col 34 Lines 50 – 54, “The illustrated embodiment of the IIMIGM system may include one or more components, not shown, to each perform portions of the functionality of the IIMIGM system, and the memory may further optionally execute one or more other programs 335 . . .” The above limitations implements the limitations of Claim 1 and is therefore rejected under a similar rationale as that of Claim 1.) Regarding Claim 8, Hutchcroft in view of Yang and Lorenzo disclose The three-dimensional reconstruction device according to claim 1, the processor further configured to execute operations comprising: displaying the three-dimensional reconstruction of the structure as output in a display of a virtual reality (VR) goggle. (See Hutchcroft Col 41 Lines 37 – 40, “provide the generated 2D floor plan and/or 3D computer model floor plan and/or other generated or determined information for display on one or more client devices . . .” See Lorenzo [0069], “In addition to creating or displaying 2D floor plans 300 and 3D models 400 . . .” Also see Lorenzo [0005], “Taking advantage of this behavior, stereoscope-style enclosures for smartphones (such as Google Cardboard viewers and the Samsung Gear VR) can be used to view 360 degree images in an immersive format similar to virtual reality. A smartphone display may be viewed through lenses contained within the enclosure, as opposed to virtual reality headsets that contain their own dedicated displays .” Here, Hutchcroft already teach to display the 3D model, and although Lorenzo doesn’t explicitly recommend VR headsets, Lorenzo does show that they are a common device used for displaying 3D models, and thus it would have been obvious to use them. The motivation to combine would have been similar that of Claim 1 rejection motivation.) Regarding Claim 9, Hutchcroft in view of Yang and Lorenzo disclose The three-dimensional reconstruction device according to claim 1, wherein the three-dimensional reconstruction of the structure represents an inspection image of the interiror of the structure by infrastructure equipment. (See Hutchcroft Col 9 Lines 65 – Col 10 Lines 5, “In particular, target panorama images 165 are indicated in FIG. 1A that have been acquired by one or more mobile computing devices 185 with imaging systems and/or by one or more separate camera devices 184 (e.g., without onboard computing capabilities), such as with respect to one or more buildings or other structures and under control of an Interior Capture and Analysis (“ICA”) system 160 . . .” Here, since the images are of the interior of the building, and the 3D model (three-dimensional reconstruction of the structure) would be based on those images, then the 3D model is be considered as representing “an inspection image of the interiror of the structure”. Also see Lorenzo [0010], “Any sort of change to building interior , building exterior, building property, or building aesthetics may be monitored . The changes may include, but is not limited to, any combination of building materials, electrical, plumbing, landscaping, HVAC, mechanical, doors or windows, access, security , environmental, privacy, decoration, or paint/surface treatments.” See Lorenzo [0011], “Another advantage of the present invention is that it allows either a conventional (non-360 degree) or a 360 degree camera or image capture device to be used . Conventional image capture devices are now ubiquitous, and allow the option of an untrained individual to capture a series of construction photographs.” Note that the although cameras and imaging devices of Hutchcroft and Lorenzo are not explicitly stated to be part of the building’s infrastructure, since Lorenzo does mention that these cameras/imaging devices can be used to monitor for any changes related to the building, this implies some permeances or persistence of the camera/imaging device within the infrastructure to be able to monitor for these changes. The persistent monitoring can broadly be interpreted as the cameras/imaging devices being considered as “infrastructure equipment”. Additionally, Lorenzo states that any conventional camera / image capture device can be used. One such conventional and common camera is a security camera, which can considered as “infrastructure equipment”. The motivation to combine would have been similar to that of Claim 1 rejection motivation.) Allowable Subject Matter 12-151-08 AIA 07-43 12-51-08 Claim 2-3, 5-6 objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims. 13-03-01 AIA The following is a statement of reasons for the indication of allowable subject matter: Regarding Claim 2, the cited prior art does not disclose or render obvious the combination of elements cited in the claims as a whole. Specifically, the cited prior art fails to disclose or render obvious the limitations: detecting a straight line in the generated plan view and specifying at least one intersection of the straight line as a first intersection; and detecting a straight line in the acquired drawing and specifying at least one intersection of the straight line as a second intersection, wherein the performing alignment further comprises performing the alignment using the first and second intersections as initial positions. Thus Claim 2 contains allowable subject matter. Regarding Claim 3, the cited prior art does not disclose or render obvious the combination of elements cited in the claims as a whole. Specifically, the cited prior art fails to disclose or render obvious the limitations: generating a plan view further comprises obtaining a distance of a camera imaging the panoramic image to a wall surface of the structure and plotting the distance at each imaging angle in a horizontal direction of the camera to generate the plan view. Thus Claim 3 contains allowable subject matter. Regarding Claim 5, Claim 5 contains similar limitations as to Claim 2 and therefore contains similar allowable subject matter. Regarding Claim 6, Claim 6 contains similar limitations as to Claim 3 and therefore contains similar allowable subject matter . Conclusion 07-96 AIA The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Reference Furukawa et al. (US 20140301633 A1) is made of record as an art that teaches obtaining panoramic images (See [0031]), acquiring the ceilings height of the building (See [0034]), using the panoramic images to generate a floorplan (See [0039] - [0042]), and generating a three-dimensional model of the interior (3D reconstruction) based on the floorplan (See [0006] and [0045]). Any inquiry concerning this communication or earlier communications from the examiner should be directed to THANG G HUYNH whose telephone number is (571)272-5432. The examiner can normally be reached Mon-Thu 7:30am-4:30pm EST | Fri 7:30am-11:30am EST. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee Tung can be reached at (571)272-7794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. /KEE M TUNG/Supervisory Patent Examiner, Art Unit 2611 /T.G.H./Examiner, Art Unit 2611 Application/Control Number: 18/866,542 Page 2 Art Unit: 2611 Application/Control Number: 18/866,542 Page 3 Art Unit: 2611 Application/Control Number: 18/866,542 Page 4 Art Unit: 2611 Application/Control Number: 18/866,542 Page 5 Art Unit: 2611 Application/Control Number: 18/866,542 Page 6 Art Unit: 2611 Application/Control Number: 18/866,542 Page 7 Art Unit: 2611 Application/Control Number: 18/866,542 Page 8 Art Unit: 2611 Application/Control Number: 18/866,542 Page 9 Art Unit: 2611 Application/Control Number: 18/866,542 Page 10 Art Unit: 2611 Application/Control Number: 18/866,542 Page 11 Art Unit: 2611 Application/Control Number: 18/866,542 Page 12 Art Unit: 2611 Application/Control Number: 18/866,542 Page 13 Art Unit: 2611 Application/Control Number: 18/866,542 Page 14 Art Unit: 2611
Prosecution Insights