DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on/after Mar. 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 USC 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 20-27 and 30-37 are rejected under 35 U.S.C. 103 as being unpatentable over Steedly et al. (U.S. PG-PUB 2009/0244062, ‘STEEDLY’) in view of Masuda et al. ("Development of '3D Measurement System' Using Images Taken with a 'FinePix REAL 3D W3' 3D Digital Camera", pub. 2012, 'MASUDA').
Regarding claim 20, STEEDLY discloses a method of scaling a … (3D) scene, the method comprising:
PNG
media_image1.png
578
422
media_image1.png
Greyscale
receiving … images comprising a building object (STEEDLY; FIG. 1; ¶ 0020; “FIG. 1 shows a collection of photos 100 [which] are of a … scene, in this case, a shed. The photos 100 were taken from different positions relative to the shed, as seen in the differences between the photos 100. Some … photos overlap. That is to say, parts of the shed in one photo are also in another photo. … overlapping photos 102/104 both show portions of a door. … Because of the overlap between photos, it is possible to reconstruct some basic [3-D] information about the photos 100 and the shed.”);
generating a multidimensional building model based on the … images (STEEDLY; FIGS. 2-3; ¶ 0021; “FIG. 2 shows a system for generating a model from a collection of photos. The system starts with the collection of photos 100. A photo-synthesizer 120 … performs a process 122 for synthesizing the photos 100 to obtain [3-D] information about the photos 100. … the photo synthesizer 120 receives the photos 100 and analyzes the photos to extract features of the real-world subject in the photos 100. Features common to overlapping photos are identified … [and] are used to determine camera postures for respective photos. These will be referred to as viewpoints or virtual cameras … A photo's virtual camera preferably lies on a line normal to the photo (preferably from the center of the photo) and at a distance known as the focal length. … see FIG. 3, showing … photos 100 of subject 150 and corresponding reconstructed [3-D] arranged virtual cameras 152, which in practice are stored as corresponding data structures/objects … The photo-synthesizer 120 reconstructs the positions and directions from which the photos were captured.” ¶ 0023; “… the photo-synthesizer 120 outputs the point cloud and camera poses/postures 122, which are made available to a model renderer/editor 124 [which] has … software module(s) for performing a process 126, which may include operations such as: maintaining and rendering a [3-D] model; using points in the point cloud and/or vanishing points in the photos as constraints for fitting planes of the [3-D] mesh model; interactively altering the geometry of the model and in particular orienting and translating planes of the model (perhaps as informed by the point clouds, photos, and virtual cameras); displaying photos for the purpose of selecting constraint points in the point cloud; marking the photos to define patches for generating corresponding textures for the model; etc. … the model renderer/editor 124 outputs a mesh model 128 which may also include textures for the model.”);
([MASUDA discloses this limitation.]);
PNG
media_image2.png
652
478
media_image2.png
Greyscale
determining a position of a camera relative to the image … (STEEDLY; FIG. 4; ¶ 0022; “The photo synthesizer 120 also determines a sparse point cloud comprised of points in [3-D] space, where the points are an estimation of the location in [3-D] space of features of the subject found in the photos. A point may be found when computing the virtual cameras. … this may involve identifying a feature 140A of the photographic subject in a photo 141, looking for the same feature 140B in another photo 142, and if it is found, simultaneously estimating the [3-D] camera positions of the photos and a position of point 140C (an example of a point in the point cloud) such that when the point 140C is re-projected back into the corresponding photo's cameras it re-projects to points 140A, 140B where the feature is found in the respective photos 141, 142.”), wherein the position is determined from the at least one boundary (STEEDLY; FIG. 7; ¶ 0031; “… photos 200 are oriented in a virtual [3-D] space according to their respective … virtual cameras 202. In a virtual [3-D] space, the virtual cameras 202 have a position and orientation, and the respective photos 200 are shown according to their focal distance from their virtual cameras 202. The photos contain image data of a real-world subject, such as a building 204. … a vanishing point 206 (and corresponding vanishing direction) may be computed from a photo 200. … each photo 200 contains a same feature 205 of the building 204. … the feature 205 is a corner of building 204. A corresponding feature point 208 of a point cloud is also shown [The Examiner asserts that a person having ordinary skill in the photogrammetric art(s) knows that a ‘feature point … of a point cloud’ represents a point (corner) of a façade, which is at least one boundary of a building structure. Furthermore, the Examiner asserts that a person having ordinary skill in the photogrammetric art(s) also knows that identifying a point on an ‘architectural surface/building/model’ is an equivalent technique to finding a relevant boundary point of a non-architectural element. The Examiner combines the teaching(s) of STEEDLY with the teaching(s) of MASUDA, which EXPLICITLY teaches the identification/detection and subsequent graphical highlighting of non-architectural structures.]. The feature point 208 is a [3-D] point where features 205 intersect when projected from their respective virtual cameras 202 and photos 200.”); and
([MASUDA discloses this limitation.]).
STEEDLY does not explicitly disclose the following limitations, which MASUDA discloses:
PNG
media_image3.png
820
566
media_image3.png
Greyscale
identifying at least one boundary of a non-architectural element in an image (MASUDA; FIG. 3; p. 40, left column; [The Examiner notes that FIG. 3 of MASUDA depicts the identification of two measured dimensions (height, width) of the boundary of the FUJIFILM sign, which is analogous to a non-architectural element in an image. MASUDA achieves this result by ‘obtaining the distance between two points or the area of a polygon specified on a plane.’]); and
PNG
media_image4.png
685
1233
media_image4.png
Greyscale
generating a scale for the multidimensional building model based on the position of the camera (MASUDA; FIG. 3; p. 40, left col.; “The system locates the 3D position of a specified point of an object in a photographic image using [its] xyz coordinates with the position of the camera as the origin.”; [The Examiner notes that, using the techniques disclosed above, dimensions of the building, specifically the height of the cutout vestibule, is measurable. The Examiner further asserts that any desired measurement of the building is then obtainable, and the overall scale of the building may be extrapolated from the various measurements. The Examiner also asserts that the FUJIFILM sign, being extrinsic {non-architectural, non-structural} to the architectural façade, likely has known physical dimensions and may be further exploited to extrapolate physical dimensional measurements of the building.]).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to modify the method of scaling a 3D scene of STEEDLY to include the identifying at least one boundary of a non-architectural element in an image and the generating a scale for the multidimensional building model based on the position of the camera of MASUDA. The motivation for this modification is to exploit prior knowledge relating to the size, shape, and/or dimensions of an extrinsic planar object (such as a rectangular sign with known height and width) that may be positioned alongside/proximate to a façade of an architectural structure. Having this a priori knowledge allows for photogrammeters to extrapolate dimensions of the proximate façade, which may have much more complicated features, such as non-polygonal elements, elements of varying depth, occluded elements, etc. Once the dimensions of the architectural façade are extrapolated, the overall scale of the structure may be determined using the dimensions of the various architectural features and façade.
Regarding claim 30, STEEDLY-MASUDA disclose … non-transitory computer-readable medium/media storing instructions that, when executed by … processor(s), cause the … processor(s) to perform a method of scaling a … (3D) scene (STEEDLY; ¶ 0042; “… features discussed above can be realized in the form of information stored in … computer … readable media. This is deemed to include … media such as optical storage (e.g., CD-ROM), magnetic media, flash ROM [etc.] The stored information can be in the form of machine executable instructions (e.g., compiled executable binary code) … that can be used to enable/configure computing devices to perform the various embodiments discussed above. This is also deemed to include at least volatile memory such as RAM and/or virtual memory storing information such as CPU instructions during execution of a program carrying out an embodiment, as well as non-volatile media storing information that allows a program or executable to be loaded and executed. The embodiments and featured can be performed on any type of computing device, including portable devices, workstations, servers, mobile wireless devices, and so on.”), the method comprising: … ([The following limitations are repeated verbatim from those recited in independent claim 20.]).
Regarding claim 21 and claim 31, STEEDLY-MASUDA disclose the method of claim 20 and the non-transitory computer-readable medium of claim 30, wherein the … images are captured by a camera (STEEDLY; FIG. 1; ¶ 0020; “The collection of photos 100 may be taken from a same camera or from different cameras. The photos 100 might also be video frames captured from a video camera.”).
Regarding claim 22 and claim 32, STEEDLY-MASUDA disclose the method of claim 20 and the non-transitory computer-readable medium of claim 30, wherein the … images are captured by a smartphone (STEEDLY; ¶ 0042; “The embodiments and featured can be performed on any type of computing device, including portable devices, workstations, servers, mobile wireless devices …”).
Regarding claim 23 and claim 33, STEEDLY-MASUDA disclose the method of claim 20 and the non-transitory computer-readable medium of claim 30, wherein the multidimensional building model comprises a 3D model (STEEDLY; FIG. 2; ¶ 0023; “… the photo-synthesizer 120 outputs the point cloud and camera poses/postures 122, which are made available to a model renderer/editor 124 [which] has … software module(s) for performing a process 126, which may include operations such as: maintaining and rendering a [3-D] model; using points in the point cloud and/or vanishing points in the photos as constraints for fitting planes of the [3-D] mesh model … The model renderer/editor 124 outputs a mesh model 128 which may also include textures for the model.”).
Regarding claim 24 and claim 34, STEEDLY-MASUDA disclose the method of claim 20 and the non-transitory computer-readable medium of claim 30, wherein the multidimensional building model comprises a 2D model (STEEDLY; FIGS. 8-9; ¶ 0035-37; “… the model 201 and plane 210 [‘2D model’], with the photos 200 projected 252 thereon, are rendered 254 and displayed. The user manually adjusts 256 the plane 210 … by using a mouse to adjust the orientation and/or the location of the plane. … the depth of the plane 210 may be adjusted. This involves the user translating the plane 210 along its normal. … when adjusting 256 the plane 210, the plane 210 can be automatically snapped to various positions and/or orientations … based on the point cloud or constraints derived from the point cloud. Plane adjustment 256 may also involve revising the border or perimeter 284 of the surface defined by the plane 210. The projecting 252, rendering 254, and adjusting 256 may be repeated until the user, according to the visual feedback of the rendering 254 and displaying, has finally arranged the plane 210 to minimize the visual artifacts and accurately model the wall that the plane 210 represents. At this point, the user might also select … photo(s) for texturing the plane 210.”).
Regarding claim 25 and claim 35, STEEDLY-MASUDA disclose the method of claim 20 and the non-transitory computer-readable medium of claim 30, further comprising instructions for calculating a measurement of the at least one boundary of the non-architectural element (MASUDA; FIG. 3; p. 40; [The width and height measurements are displayed alongside the boundaries of the rectangular FUJIFILM sign.]), and wherein the position of the camera is based on the measurement (MASUDA; FIG. 3; p. 40, left col.; “The system locates the 3D position of a specified point of an object in a photographic image using [its] xyz coordinates with the position of the camera as the origin. Using these coordinates, the system can obtain the distance between two points or the area of a polygon specified on a plane.”).
Regarding claim 26 and claim 36, STEEDLY-MASUDA disclose the method of claim 25 and the non-transitory computer-readable medium of claim 35, wherein the measurement comprises a dimensional ratio (MASUDA; FIG. 3; p. 40; [The width and height measurements are displayed alongside the boundaries of the rectangular FUJIFILM sign. The Examiner asserts that the combined knowledge of the width and height of the sign constitutes a ratio.]).
Regarding claim 27 and claim 37, STEEDLY-MASUDA disclose the method of claim 20 and the non-transitory computer-readable medium of claim 30, wherein at least one boundary of the non-architectural element comprises a dimension reference (MASUDA; p. 40; FIG. 3, [The FUJIFILM rectangular sign, which is extrinsic to the architectural structure, has dimension references along both width and height boundaries.]).
Claims 28-29 and 38-39 are rejected under 35 U.S.C. 103 as being unpatentable over STEEDLY in view of MASUDA as applied to claims 20 and 30 above, respectively, and further in view of Baker (U.S. PG-PUB 2015/0325038, 'BAKER').
Regarding claim 28 and claim 38, STEEDLY-MASUDA disclose the method of claim 20 and the non-transitory computer-readable medium of claim 30; however, STEEDLY-MASUDA do not explicitly disclose that generating the multidimensional building model comprises applying the scale to the multidimensional building model, which BAKER discloses (BAKER; FIG. 1; ¶ 0065; “Each stereoscopic-3D image of an object has an image resolution based on the specific image capture device used to acquire the images and the object's distance from the image capture device. This is a known principle of optical image acquisition. Similarly, for each planar surface and object in an image of a customer's space, there is a corresponding resolution based on the image capture device and the surface/object distance from the image capture device. To ensure appropriate realism and make the inserted object appear to be part of the original image, the scale of the perspective view of the object is first adjusted to match its correct size for the scale of the location into which it is inserted. It is then further adjusted to match the resolution for the area into which it is being inserted, which takes into account its distance from and the perspective of the image capture device. These actions are shown as step 126. This is readily accomplished because the dimensions of the space and all associated objects are recorded in the 3D model of the space. The result of step 126 is a properly-scaled and resolution-matched stereoscopic-3D photographic interpolated image of a selected physical object. Once the stereoscopic-3D image of the object is thus adjusted, it is embedded in the stereoscopic-3D images of the real space as shown in 128.”).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to modify the method of claim 20 and the non-transitory computer-readable medium of claim 30 of STEEDLY-MASUDA to include the applying the scale to the multidimensional building model of BAKER. The motivation for this modification is to achieve a properly-scaled and resolution-matched stereoscopic-3D photographic interpolated image of a 3-D architectural structure (BAKER; ¶ 0065).
Regarding claim 29 and claim 39, STEEDLY-MASUDA disclose the method of claim 20 and the non-transitory computer-readable medium of claim 30; however, STEEDLY-MASUDA do not explicitly disclose applying the scale to the multidimensional building model, which BAKER discloses (BAKER; FIG. 1; ¶ 0065; [See analyses for the rejection(s) of claim 28 and 38 in the Office action above.]).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to modify the method of claim 20 and the non-transitory computer-readable medium of claim 30 of STEEDLY-MASUDA to include the applying the scale to the multidimensional building model of BAKER. The motivation for this modification is to achieve a properly-scaled and resolution-matched stereoscopic-3D photographic interpolated image of a 3-D architectural structure (BAKER; ¶ 0065).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Dai et al. (“Assessing the Accuracy of Applying Photogrammetry to Take Geometric Measurements on Building Products”, published Feb. 2010) disclose techniques for using photogrammetry to assess the accuracy of measurements of building construction components. Photogrammetric measurements are compared to tape measurements, and statistical confidence intervals are created.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JONATHAN M COFINO whose telephone number is (303) 297-4268. The examiner can normally be reached Monday-Friday 10A-4P MT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kent Chang can be reached at 571-272-7667. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JONATHAN M COFINO/Examiner, Art Unit 2614
/KENT W CHANG/Supervisory Patent Examiner, Art Unit 2614
Prosecution Insights