Detailed Action
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The amended claims and associated arguments/ remarks filed on 4/8/2025 were received and considered.
Claims 1, 11, 42, and 67 have been amended.
Claims 1-4, 11, 15, 16, 26, 27, 30, 32, 37, 40, 42, 44, 46, 57, 67, 71, and 73 are pending.
Claim Objections
Claim 42 is objected to because of the following informalities: the claim has been mislabeled as “Previously Presented”. The claim has been amended and should be labeled as “Currently Amended”. Appropriate correction is required.
Response to Arguments
Applicant’s arguments, see Remarks, filed 4/8/2025, with respect to the rejection(s) of claim(s) 1-4, 11, 15, 16, 26, 27, 30, 32, 37, 40, 42, 44, 46, 57, 67, 71, and 73 under USC 103 have been fully considered. However, upon further consideration, a new ground(s) of rejection is made in view of Zhang et al. (US 7929751 B2).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-3, 15, 16, 37, 42, 44, 57, and 67 are rejected under 35 U.S.C. 103 as being unpatentable over Zhang et al. (US 7929751 B2) referred to as Zhang hereinafter and further in view of Wang (US 20150070473 A1).
Regarding claim 1, Zhang teaches A computer implemented system for three dimensional scanning comprising: (“Real-time 3D scanning” Zhang, col. 2, lines 20-21) a projector configured to project structured light onto a three-dimensional object; (“projected by structured light onto an object” Zhang, col. 3, lines 62-63)
a camera configured to capture fringe images of the object, (“The captured fringe images 312B and 314B” Zhang, col. 13, lines 53) each image comprising a plurality of pixels; (“for each pixel of the captured image, the corresponding pixel of the projected image is known” Zhang, col. 12, lines 24-26) and (“Because the projected fringe images are composed of uniform stripes, and assuming that a fringe image has a fringe pitch P, which is the number of pixels per fringe period, and the total number of pixels in the x direction” Zhang, col. 21, lines 61-64)
a processor configured to process the fringe images to extract a phase map and a texture image, (“capturing images of reflections of the incident patterns as distorted by the surface of the object, and processing the reflected images to extract the phase of each of their pixels while removing phase discontinuities (a process called "phase unwrapping").” Zhang, col. 1, lines 44-48) and (“phase unwrapping 2418 produces an image 2408, after which geometry is extracted at 2420 to produce geometry image 2410.” Zhang, col. 27, lines 1-4)
wherein the texture image is determined for each pixel as the sum of an average intensity value of the pixel and a modulation value of the pixel,
PNG
media_image1.png
227
470
media_image1.png
Greyscale
Zhang, col. 14, lines 4-15
to calculate depth information from the phase map, and to perform 3D surface reconstruction based on the depth information and texture image, (“reconstructing the geometry of the object from reflections therefrom; projecting a plurality of fringe patterns onto the object at a second, relatively higher spatial frequency and reconstructing the geometry of the object from reflections therefrom, said reconstruction including pixels of ambiguous phase;” Zhang, col. 8, lines 5-10) and (“(“The capturing of phase distortions permits recovery of the target object's geometry--i.e., surface depth/contouring.” Zhang, col. 10, lines 51-52)
wherein the processor generates the phase map based on consideration of an intensity bias component of the fringe images, a modulation component of the fringe images, and an unwrapped phase of the fringe images. (“Referring again to FIG. 7, further processing of wrapped phase image 721 includes employing a multi-level quality guided phase unwrapping algorithm 722 to generate an unwrapped phase image 725 at 724. Markers encoded at 708A-708C are detected at 730 in order to help generate absolute phase map 727 at 726, which is followed by determination of absolute coordinates 729 at 728.” Zhang, col. 14, lines 40-46)
However, Zhang does not teach to calculate depth information from the phase map,
Wang teaches to calculate depth information from the phase map, (“a phase map, having values of .phi.(x, y) for all coordinates over the distorted fringe image, is used to calculate the depth distribution of the object's shape.” Wang, para. [0004])
Zhang and Wang are combinable because they are from the same field of endeavor, image processing.
At the time of the invention was filed, it would have been obvious to a person of ordinary skill in the art to modify Zhang in light of Wang’s calculating depth information from the phase map. One would have been motivated to do so because it can improve design of a fringe pattern over the teachings of SU such that the improved design enables the fringe order to be computed while the measurement accuracy can be maintained. (Wang, para. [0006])
Regarding claim 2, Wang teaches wherein the structured light comprises a plurality of phase lines, and each phase line is distorted to a curve on the three dimensional object. (“the present invention is not limited to using a sinusoidal function in modulating the light intensity. A non-sinusoidal function may be used. In case that more-than-one predetermined fringe patterns are used, such plural fringe patterns may be derived from one reference fringe pattern with different phase offsets for different predetermined fringe patterns, each of the phase offsets denoting a shift in position for fringes in the reference fringe pattern.” Wang, para. [0028]) and (“the object distorts the one or more predetermined fringe patterns, thereby generating one or more distorted fringe images. The method further comprises analyzing the one or more distorted fringe images to estimate an unwrapped phase map that characterizes one or more intensity distributions of the one or more distorted fringe images. A depth distribution of the object's shape is hence obtainable from the unwrapped phase map.” Wang, para. [0008])
Regarding claim 3, Zhang teaches wherein the image is captured by a first camera for capturing the fringe images of the three-dimensional object, (“a method is provided of capturing an image of a 3D object which encodes the surface geometry of at least a portion of the object, comprising: projecting a plurality of fringe patterns onto the object; capturing phase alterations in reflections of the fringe patterns from the object;” Zhang, col. 7, lines 52-57) and a second camera for capturing a color texture image of the object. (“The images of the reflections of the patterns also may be captured in a black and white mode and in a color mode.” Zhang, col. 6, lines 2-4)
Regarding claim 15, Zhang teaches wherein a quality map and a mask of a facial skin area is generated by the texture image, and the quality map and mask is inputted into a phase unwrapping algorithm by the processor to determine an unwrapped phase. (“FIG. 15 shows an example of applying a three-level quality-guided phase unwrapping algorithm to a wrapped image of a human face. The top row shows the unwrapped data sets 1500, 1502 and 1504 after each of the three unwrapping levels and the bottom row shows the corresponding images 1506, 1508 and 1510. It can be seen that the main features of the face are unwrapped already in level 1. In level 2, the image areas corresponding to some of the missed features, for example, the right part of the chin, are unwrapped. The remaining details, such as the left ear and part of the neck, are obtained in the last, third, level.” Zhang, col. 19, lines 42-52)
Regarding claim 16, Zhang teaches wherein the processor transforms world coordinates of a point to camera coordinates, transforms the camera coordinates to camera projective coordinates, and transforms the camera projective coordinates to distorted camera projective coordinates.
PNG
media_image2.png
431
453
media_image2.png
Greyscale
Zhang, col. 21, lines 40-60
Regarding claim 37, Zhang teaches wherein textural features and the geometric features are extracted from the first and second fringe images. (“It is desirable to extract the geometric texture from a real surface with complicated profiles and transfer the texture to a geometric model. By using holoimages, geometric textures can be extracted from real objects easily.” Zhang, col. 28, lines 25-29)
Regarding claim 42, refer to the explanation of claim 1.
Regarding claim 44, refer to the explanation of claim 3.
Regarding claim 57, refer to the explanation of claim 16.
Regarding claim 67, refer to the explanation of claims 1 and 26.
Claim 4 and 46 are rejected under 35 U.S.C. 103 as being unpatentable over Zhang and Wang as mentioned above and further in view of Hicks (US 10627512 B1).
Regarding claim 4, the combination of Zhang and Wang does not teach wherein exposure cycles of both the first and second cameras are synchronized and wherein the first camera is triggered to capture an image on each off cycle, and the second camera is triggered to capture an image every three off cycles.
However, Hicks teaches wherein exposure cycles of both the first and second cameras are synchronized and wherein the first camera is triggered to capture an image on each off cycle, and the second camera is triggered to capture an image every three off cycles. (“The first camera frame capture can be synchronized with the start of the lidar scan. The third camera frame capture will then be completed at about the time of the end of the lidar scan.” Hicks, col. 9, lines 64-67)
Zhang, Wang, and Hicks are combinable because they are from the same filed of endeavor, image processing.
At the time of the invention was filed, it would have been obvious to a person of ordinary skill in the art to modify Zhang and Wang in light of Hicks’s synchronizing first and second camera. One would have been motivated to do so because it can provide more accurate results. (Hicks, col. 4, lines 3-4)
Regarding claim 46, refer to the explanation of claim 4.
Claim(s) 26 and 27 are rejected under 35 U.S.C. 103 as being unpatentable over Zhang and Wang as mentioned above and further in view of Ji (US 20210118213 A1).
Regarding claim 26, the combination of Zhang and Wang does not teach wherein at least one point cloud is generated based on the depth information by the processor and the point cloud is processed by the processor to form a triangle mesh; wherein the processor performs conformal geometry methods for image and shape analysis and real-time tracking applications.
However, Ji teaches wherein at least one point cloud is generated based on the depth information by the processor and the point cloud is processed by the processor to form a triangle mesh; (“A point cloud may be a set of data points in space which measure a large number of points on the external surface of the object. The point cloud of an object may be converted, by a computer system, into a polygon mesh or triangle mesh through the process of surface reconstruction.” Ji, para. [0038])
wherein the processor performs conformal geometry methods for image and shape analysis and real-time tracking applications. (“At 525, the computer system reconstructs, based at least in part on the depth data, 3D object geometries. For each rotation, the computer system may reconstruct a 3D geometry of a scanned object. The 3D geometry may be determined by various means, including utilizing depth maps and/or point clouds. For example, the computer system may determine a depth map of a scanned object at a first rotation, a depth map of a scanned object at a second rotation, and so forth. In such an example, each depth map at each rotation may be a 3D geometry.” Ji, para. [0056])
Zhang , Wang, and Ji are combinable because they are from the same field of endeavor, image processing.
At the time of the invention was filed, it would have been obvious to a person of ordinary skill in the art to modify Zhang and Wang in light of Ji’s generating a point cloud. One would have been motivated to do so because it can result in increased pixel approximation. (Ji, para. [0049])
Regarding claim 27, Ji teaches wherein the processor is further configured to use ambient, modulation and projector parameters to estimate surface normal information in the process of generating at least one point cloud. (“Other approaches convert the point cloud into a volumetric distance field and reconstruct the implicit surface so defined through a marching cubes algorithm.” Ji, para. [0038])
Claim(s) 30, 32, 71, and 73 are rejected under 35 U.S.C. 103 as being unpatentable over Zhang and Wang as mentioned above and further in view of Bronstein et al. (US 20170089690 A1) referred to as Bronstein hereinafter.
Regarding claim 30, the combination of Zhang and Wang does not teach wherein an image is captured from two different viewing angles to obtain stereoscopic depth information; and wherein the processor is configured to use the stereoscopic depth information as an input into the generation of at least one point cloud.
However, Bronstein teaches wherein an image is captured from two different viewing angles to obtain stereoscopic depth information; (“Capture device 100 may utilize two or more physically separated cameras that may view a capture area from different angles, to obtain visual stereo data that may be resolved to generate depth information.” Bronstein, para. [0029])
and wherein the processor is configured to use the stereoscopic depth information as an input into the generation of at least one point cloud. (“Using the results of reconstruction, processing logic re-projects the first reconstructed point to obtain a second point identified by a pair of coordinates (processing block 403). In one embodiment, re-projecting the first reconstructed point to obtain a second point identified by a pair of coordinates comprises re-projecting the 3D point location on the projector plane using a predefined projection matrix of the projector.” Bronstein, para. [0058])
Zhang, Wang, and Bronstein are combinable because they are from the same filed of endeavor, image processing.
At the time of the invention was filed, it would have been obvious to a person of ordinary skill in the art to modify Zhang and Wang in light of Bronstein’s two different viewing angles. One would have been motivated to do so to perform triangulation accurately in the devices. (Bronstein, para. [0003])
Regarding claim 32, Zhang teaches wherein first fringe images are captured at a first time and used to perform a first 3D surface reconstruction by the processor, (“A projector, which in presently preferred embodiments of the invention is a single-chip DLP projector (e.g., PLUS U5-632h, PLUS Vision Corp.), receives the color coded images from the signal generator 302, as shown in FIG. 8, and projects them in a B/W mode sequentially and repeatedly at a frame rate of 120 fps. A high-speed CCD camera (e.g., Pulnix TM-6740CL, JAI PULNiX, Inc., San Jose, Calif.) synchronized with the projector is used to capture the reflected patterns at a speed of 180 fps. Based on the 2+1 phase-shifting algorithm, any three successive images can be used to reconstruct the 3D geometry.” Zhang, col. 16, lines 15-25)
Bronstein teaches second fringe images are captured at a second time and used to perform a second 3D reconstruction by the processor, and the first and second 3D reconstructions are registered for comparison. (“performing reconstruction based on a coordinate of the undistorted camera location and the new distorted projector location coordinate to produce a second reconstructed point comprises determining a 3D point location by applying the triangulation-based reconstruction to the distortion-corrected 1D projector code from estimating the new distorted projector location coordinate.” Bronstein, para. [0060])
Regarding claim 71, refer to the explanation of claim 30.
Regarding claim 73, refer to the explanation of claim 32.
Claim 40 is rejected under 35 U.S.C. 103 as being unpatentable over Zhang and Wang as mentioned above and further in view of Wu et al. (US 20200349729 A1) referred to as Wu hereinafter.
Regarding claim 40, the combination of Zhang and Wang does not teach wherein the processor is further configured to model a phase-height map as a polynomial function at each pixel the camera, wherein the processor is further configured to estimate coefficients of the polynomial in a camera-projector calibration process using an optimization algorithm.
However, Wu teaches wherein the processor is further configured to model a phase-height map as a polynomial function at each pixel the camera, (“Referring to FIG. 1, the optical layer 106 with a phase mask height map 112 as learnable parameter, a physics-based model first simulates depth dependent PSFs 114, and then applies these PSFs to red green blue depth (RGB-D) input to formulate the coded image on the sensor.” Wu, para. [0041]) wherein the processor is further configured to estimate coefficients of the polynomial in a camera-projector calibration process using an optimization algorithm. (“Although the depth-dependent PSF response of the phase mask is known from simulation, the prototype camera in one or more embodiments calibrated to account for any mismatch born out of physical implementation such as aberrations in fabricated phase mask and phase mask aperture alignment. An optimization based approach adopted where the PSFs from a set of sharp and coded image pairs of a calibration pattern are estimated.” Wu, para. [0117])
PNG
media_image3.png
32
457
media_image3.png
Greyscale
Zhang, Wang, and Wu are combinable because they are from the same filed of endeavor, image processing.
At the time of the invention was filed, it would have been obvious to a person of ordinary skill in the art to modify Zhang and Wang in light of Wu’s modeling a phase-height map as a polynomial function. One would have been motivated to do so because it can improve the performance of the depth reconstruction network. (Wu, para. [0123])
Allowable Subject Matter
Claim 11 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
Applicant’s amendment necessitated new ground(s) of rejection presented in this Office Action.
THIS ACTION IS MADE FINAL. See generally MPEP § 706.02(l)(3). Applicant is reminded of the
extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from
the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date
of this final action and the advisory action is not mailed until after the end of the THREE-MONTH
shortened statutory period, then the shortened statutory period will expire on the date the advisory
action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing
date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX
MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PARDIS SOHRABY whose telephone number is (571)270-0809. The examiner can normally be reached Monday - Friday 9 am till 6pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jennifer Mehmood can be reached at 5712722976. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PARDIS SOHRABY/ Examiner, Art Unit 2667
/JENNIFER MEHMOOD/Supervisory Patent Examiner, Art Unit 2671
Prosecution Insights