DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
1. The information disclosure statements (IDS) submitted on 2/10/2025, 12/5/2025, 2/9/2026, 5/28/2026, and 9/16/2026 are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statements have been considered by the examiner.
Drawings
2. The drawings are objected to as failing to comply with 37 CFR 1.84(p)(5) because they do not include the following reference sign(s) mentioned in the description: 103 of [0070], line 5. Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Specification
3. The disclosure is objected to because of the following informalities:
In [0003], line 11, "scene in time sequence," should read "scene in a time sequence,"
In [0031], line 13, "is only schematic," should read "is only a schematic,"
In [0042], line 23, "objects is difficultly to realize." should read "objects is difficult to realize."
In [0070], line 5, "plurality of image acquisition devices 103," should read "plurality of image acquisition devices 603,"
Appropriate correction is required.
Claim Objections
4. Claims 2, objected to because of the following informalities:
In claim 2, line 9, "determining the number" should read "determining a number"
In claim 4, line 2, "plurality of the second" should read "plurality of second"
In claim 5, line 2, "corresponding to the any code" should read "corresponding to any code"
In claim 7, lines 1-2, "determining the target code element set" should read "determining a target code element set"
In claim 17, line 5, "determine the number" should read "determine a number"
Appropriate correction is required.
Claim Rejections - 35 USC § 112
5. The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
6. Claims 1, 9 rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
7. Where applicant acts as his or her own lexicographer to specifically define a term of a claim contrary to its ordinary meaning, the written description must clearly redefine the claim term and set forth the uncommon definition so as to put one reasonably skilled in the art on notice that the applicant intended to so redefine that claim term. Process Control Corp. v. HydReclaim Corp., 190 F.3d 1350, 1357, 52 USPQ2d 1029, 1033 (Fed. Cir. 1999).
As described in [0003] in applicant’s specification, the technical field of the applicant's invention is directed toward: "Structured light three-dimensional reconstruction technology is a three-dimensional reconstruction technology which projects an optical coding pattern onto a surface of a measured object and recovers three-dimensional data of the surface of the object through an acquired deformation pattern." The phrase “recover[ing] three-dimensional data of the surface of the object” implies that data about the object is being actively generated and/or captured through the three-dimensional reconstruction process and is not previously known. However, the phrase “measured object” used in claims 1, 9-10, 13, and 23 and the specification is interpreted in its plain English meaning as an “object that has been measured in the past” due to “measured” being understood as a past tense version of the word “measure.” The past tense nature of “measured object” appears to temporally contradict the present tense of “recovers three-dimensional data.” Thus, “measured object” is indefinite due to the meaning being unclear. It Is believed however that the Chinese translation to English would be better characterized as “object under measurement” or “object to be measured”, representing present or future tenses respectively. This would be understood in its plain English meaning as an “object that is currently being measured or will be measured.” This appears to better align with the phrase “recovers three-dimensional data.” and the objectives of applicant’s invention. Thus, for examination, the phrase “measured object” will be understood as “object under measurement” or “object to be measured.” Applicant should consider all instances of the phrases “measured object” or “measured scene” and make appropriate corrections to address the temporal contradiction.
8. Claim 10 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being incomplete for omitting essential elements, such omission amounting to a gap between the elements. See MPEP § 2172.01. Claim 10 states the following limitation: “an image acquisition device, a projection device, and a second processor, wherein the projection device is configured to project a first image onto a surface of a measured object, wherein the first image comprises a in a preset direction, and the code elements are line segment stripes.” The last part of this limitation: “… wherein the first image comprises a [omitted details] in a preset direction, and the code elements are line segment stripes.” omits details on what the first image comprises. Based on claim 1, it is believed that the limitation should read: “wherein the first image comprises a plurality of code elements which are randomly distributed in a preset direction, and the code elements are line segment stripes” and this interpretation will be used for examination. Appropriate correction is required.
Claim Rejections - 35 USC § 103
9. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
10. Claims 1-7, 9-10, 13, and 15-23 are rejected under 35 U.S.C. 103 as being unpatentable over Huang et al. (CN-111336949-A, hereinafter "Huang") in view of Nishina et al. (JP-2017/053812-A, hereinafter "Huang"), and further in view of Hansen et al. (WO-2018/171851-A1, hereinafter "Hansen"). (Examiner’s note: Citations to Huang use the original CN-111336949-A document locations.)
11. As per claim 1, Huang discloses: A method for three-dimensional reconstruction, comprising: (Huang. page 6, [0011]-[0018], “Aiming at the problems of low robustness, poor accuracy, large image space required for decoding, and poor scalability of traditional spatial coding structured light, the present invention provides a spatial coding structured light three-dimensional scanning method. The process includes the following steps: ... S7, reconstruct the point cloud according to the image coordinates of the decoded point and the encrypted point and the corresponding projector column number.”)
projecting a first image onto a surface of a measured object (Huang. page 6, [0013], “S2: Project the pre-encoded single-frame spatial encoding pattern onto the surface of the object to be measured through a projector;”) to obtain a second image, (Huang, page 6, [0013], “S3, the camera shoots the spatial coding pattern modulated by the object, and extracts the image decoding points and encryption points;”) wherein the first image comprises a plurality of code elements (Huang, page 6, [0012], “S1, design a single-frame spatial coding pattern, including decoding points for decoding and encryption points for encryption;” and page 5, [0003], “The most important thing in the structured light method is the design and identification of codes. As shown in Figure 2, according to the different coding methods, it can be roughly divided into spatial coding, time coding, space-time coding, multi-wavelength composite coding, direct coding and line structure. Six types of light scanning. Spatial coding technology hides the coding information in a single frame of projected image space; time coding technology projects multiple different coding patterns in time sequence, and combines corresponding coded image sequences for decoding; space-time coding technology combines temporal coding schemes and space ... Light characteristics (such as color change, gray level change, etc.), directly set a code word for each pixel of the coding pattern; line structured light scanning technology projects multiple line lasers on the surface of the object to be measured, and the algorithm extracts the line structure characteristics of the image to achieve Match and rebuild.”) which are randomly distributed in a preset direction, and the code elements are [[line segment stripes;]] (Huang, page 8, [0020], “S11, the design of the basic code element. ... Assuming that the image space of the projector occupied by each code element of a single frame space encoding is the size of Grid, in the present invention, Grid=8 pixels, the spacing between the code elements along the row and column directions is Grid, and the decoding point code element is recorded in the projection” and page 8-9, [0021], “S12, the design of adjacent code element arrangement. Use the horizontal core line constraint and the relative displacement of the symbol neighborhood to encode. ... The neighborhood symbol is located in the current symbol in the vertical direction Above, it is marked as -1; the neighborhood code element is vertically flush with the current code element, then it is marked as 0; the neighborhood code element is vertically below the current code element, it is marked as 1; according to the code element The relative displacement relationship of the neighborhood, there are 9 cases in total, the upper and lower misalignment is recorded as the D value; when the spatial coding pattern is generated, the upper and lower misalignment of each column is randomly generated ...” and page 5, [0003], “As shown in Figure 2, according to the different coding methods, it can be roughly divided into spatial coding, time coding, space-time coding, multi-wavelength composite coding, direct coding and line structure.”)
determining, in the first image, a plurality of candidate code elements corresponding to any code element in the second image, (Huang, page 6, [0013]-[0014], “S3, the camera shoots the spatial coding pattern modulated by the object, and extracts the image decoding points and encryption points; S4: Determine the candidate projector column number of the decoding point by using the relationship between the horizontal epipolar line constraint and the relative displacement of the neighborhood; …” and page 7, [0025], “S31: The camera shoots the spatial coding pattern I after object modulation from different angles.” and page 7, [0029], “S35, precise extraction of encrypted points. The Gaussian ellipse fitting method is used to extract the coordinates of the entire pixel coordinates of the image encryption point to obtain the sub-pixel coordinates (u,v) of the image encryption point, then the image encryption point is recorded as Ti(x,y,u,v), 0 ≤ i ≤ t.”) and determining, from the plurality of candidate code elements, a target code element corresponding to the any code element to complete matching between the code elements in the first image and the code elements in the second image, (Huang, page 9, [0042]-[0046], “A spatial coding structured light three-dimensional scanning method and system of the present invention has the following beneficial effects compared with the prior art: ... 4)Since dots are used as the basic code element, the design of the coding pattern facilitates the extraction of high-precision matching points;” and page 7, [0029], “S35, precise extraction of encrypted points. The Gaussian ellipse fitting method is used to extract the coordinates of the entire pixel coordinates of the image encryption point to obtain the sub-pixel coordinates (u,v) of the image encryption point, then the image encryption point is recorded as Ti(x,y,u,v), 0 ≤ i ≤ t.” and page 7-8, [0031]-[0034], “S41: Determine the candidate projector column number of the decoding point by using the horizontal epipolar line constraint relationship. According to the calibration parameters of the projector and the camera, the sub-pixel image coordinates (u, v) of the image decoding point Si (x, y, u, v) are converted into camera epipolar image coordinates, and the row coordinate value of the epipolar image coordinates ... In S42, the candidate projector column number of the decoding point is selected by using the relative displacement relationship of the neighborhood. ... S51, construct the reference projector column number of the decoding point.”) wherein feature points of the target code element and feature points of the candidate code elements are on a same epipolar line; and (Huang, page 6, [0015], “S4: Determine the candidate projector column number of the decoding point by using the relationship between the horizontal epipolar line constraint and the relative displacement of the neighborhood;” and page 6, [0021], “S12, the design of adjacent code element arrangement. Use the horizontal core line constraint and the relative displacement of the symbol neighborhood to encode. The horizontal epipolar line constraint is determined by the installation of the camera and the projector. By pre-calibrating the calibration parameters of the projector and the camera, the projector image coordinates of each code point on the projector are converted into the projector’s epipolar image coordinates.” and page 7, [0031], “S41: Determine the candidate projector column number of the decoding point by using the horizontal epipolar line constraint relationship. According to the calibration parameters of the projector and the camera, the sub-pixel image coordinates (u, v) of the image decoding point Si (x, y, u, v) are converted into camera epipolar image coordinates, and the row coordinate value of the epipolar image coordinates”)
determining three-dimensional coordinates of the code elements in the second image based on a matching relationship between the code elements in the first image and the second image (Huang, page 6, [0018], “S7, reconstruct the point cloud according to the image coordinates of the decoded point and the encrypted point and the corresponding projector column number.” and page 6, [0013], “S3, the camera shoots the spatial coding pattern modulated by the object, and extracts the image decoding points and encryption points;” and page 9, [0042]-[0046], “A spatial coding structured light three-dimensional scanning method and system of the present invention has the following beneficial effects compared with the prior art: ... 4)Since dots are used as the basic code element, the design of the coding pattern facilitates the extraction of high-precision matching points;”) [[and predetermined three-dimensional coordinates of the code elements in the first image,]] and determining three-dimensional coordinates of the measured object based on the [[three-dimensional coordinates of the code elements in the first image]] and the second image to complete three-dimensional reconstruction of the measured object. (Huang, page 6, [0018], “S7, reconstruct the point cloud according to the image coordinates of the decoded point and the encrypted point and the corresponding projector column number.” and page 6, [0013], “S3, the camera shoots the spatial coding pattern modulated by the object, and extracts the image decoding points and encryption points;” and page 9, [0042]-[0046], “A spatial coding structured light three-dimensional scanning method and system of the present invention has the following beneficial effects compared with the prior art: ... 4)Since dots are used as the basic code element, the design of the coding pattern facilitates the extraction of high-precision matching points;”)
12. Huang doesn't explicitly disclose but Nishina discloses: [[projecting a first image onto a surface of a measured object to obtain a second image, wherein the first image comprises a plurality of code elements which are randomly distributed in a preset direction, and the code elements are]] line segment stripes; (Nishina, [0001], “The present invention relates to a three-dimensional measurement technique for measuring the three-dimensional position of an object by projecting patterned light, and more particularly to the determination of a line pattern in a three-dimensional measurement technique that projects a line pattern having random cuts.” and [0002], “Active stereometry is a known method for measuring the three-dimensional shape of an object. In active stereo metering, an encoded pattern light (structured light) is projected onto the object to be measured, and an image is taken.” and [0003], “A line pattern with random cuts is a pattern in which multiple parallel lines (line segments) are cut at random positions and intervals, dividing them into multiple line segments. For simplicity, in this specification, a line pattern having random cuts will be simply referred to as a "line pattern." If the epipolar line is horizontal (horizontal direction, X direction), then each line in the line pattern is generally considered to be parallel to the vertical direction (vertical direction, Y direction).”)
13. Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to modify the method of Huang to include the disclosure of line segment stripes, of Nishina. The motivation for this modification could have been to provide an alternative method of projecting code elements onto an object and provide a method for accurate 3D reconstruction.
14. Huang in view of Nishina doesn't explicitly disclose but Hansen discloses: [[determining three-dimensional coordinates of the code elements in the second image based on a matching relationship between the code elements in the first image and the second image]] and predetermined three-dimensional coordinates of the code elements in the first image, (Hansen, page 47, lines 4-18, “In an embodiment the computer system is configured for receiving and storing reference structured light data set representing the structured light beam including the set of light features. The computer system is preferably configured for receiving the reference structured light data set via a calibration step or via a user interface. ... The calibration is preferably performed by arranging the projector of the structured light arrangement and image acquisition unit(s) of the image acquisition device in predefined spatially positions relative to a plan surface, projecting the structured light to impinge the plan surface and acquiring reflected light by the image acquisition device. Thereby the computer system may also determine and store data representing the angle of divergence of the structured light beam.” and page 47, lines 23-25, “Advantageously the computer system is configured for estimating the spatial position of the projector device based on the matches between the primary features and corresponding features of the reference structured light data set.” and page 49, lines 16-21, “In an embodiment the 3D reconstruction system comprises a sensor arrangement for determining the position and orientation of the image acquisition device e.g. relative to the projector device and/or relative to a robot. In an embodiment the computer is configured for calibrating the position and orientation of the image acquisition device relative to the projector device.”; Examiner’s note: By calibrating the 3d reconstruction system with reference structured light, not only does this help determine the position and orientation of the image acquisition deice and projector device, this also determines known coordinates for the structured light during projection. The coordinates can then be used to match with coordinates determined from the structured light to perform 3D reconstruction.) [[and determining three-dimensional coordinates of the measured object based on the]] three-dimensional coordinates of the code elements in the first image [[and the second image to complete three-dimensional reconstruction of the measured object.]] (Hansen, Abstract, “A 3D reconstruction system for performing a determination of a tissue field is disclosed. The system has a structured light arrangement with a projector device configured to project a structured light beam onto at least a section of the tissue field, an image acquisition device configured for acquiring frames comprising reflections of the structured light beam from the tissue field, each frame comprises a set of pixel data associated with a time attribute, and a computer system. The projected structured light beam is stored as reference structured light data set comprising data representing a set of light features. The computer is configured for • in real time receiving frames acquired by the image acquisition device, • recognizing a plurality of light features including primary light features of the set of light features from the received set of pixel data, • matching the recognized primary light features with corresponding light features of the projected structured light beam and based on the matches estimating the spatial position of the projector device relative to at least a part of the tissue field, and • performing at least one determination of the tissue field based on the spatial position of the projector device and the recognized light features.” and page 15, line 23-page 16, line 7, “According to an embodiment of the invention it has been found that by matching data representing primary light features from the image of the structured light impinging onto the tissue field with corresponding light features of the reference structured light data set representing the projected structured light beam a very accurate determination of the spatial position of the projector device may be obtained using for example algorithm based on geometrical math. The computer system comprises the reference structured light data set and comprises an algorithm that from the matched primary light features and their orientation and optionally distortion of primary recognized features may determine the triangular configuration between the projector device, the image acquisition device and the tissue field and thus based on this perform 3D determinations of the tissue field, such as 3D distances and/or topographical configurations of the tissue field.”)
15. Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to modify the method of Huang in view of Nishina to include the disclosure of predetermined three-dimensional coordinates of the code elements in the first image, and determining three-dimensional coordinates of the measured object based on the three-dimensional coordinates of the code elements in the first image and the second image to complete three-dimensional reconstruction of the measured object, of Hansen. The motivation for this modification could have been to utilize the predetermined coordinates to not only help calibrate the reconstruction system but also to provide more accurate 3D reconstruction output.
16. As per claim 2, Huang in view of Nishina, and further in view of Hansen discloses: The method as claimed in claim 1, wherein determining, from the plurality of candidate code elements, the target code element corresponding to the any code element to complete matching between the code elements in the first image and the code elements in the second image comprises: (See rejection for claim 1.)
determining a first neighborhood code element set of the any code element in the second image and a plurality of second neighborhood code element sets of the plurality of candidate code elements; (Huang, page 8, [0032], “In S42, the candidate projector column number of the decoding point is selected by using the relative displacement relationship of the neighborhood. For the i-th decoding point Si(x,y,u,v), search for the left and right neighborhood decoding points, and obtain the corresponding D value according to the relative displacement relationship of the symbol neighborhood, which is marked as di here; Candidate projector column number CANDi(c1,...,ck) of i decoding points, get the corresponding D value according to the candidate projector column number ck, here is recorded as Dk, when di=Dk, keep this candidate projection Column number, otherwise, delete; remember that the candidate projector column number of the i-th decoding point selected through the neighborhood relative displacement relationship is CANDi(c1,...,cn), where n is the selected candidate projector The number of column numbers, (c1,...,cn) is the sequence of column number values of candidate projectors; the above process continues until the deletion of candidate projector column numbers for all image decoding points is completed; if the i-th decoding point If the D value cannot be obtained, do not do this step, and directly keep all possible candidate projector column numbers.”)
determining the number of matching neighborhood code elements in the plurality of second neighborhood code element sets with neighborhood code elements in the first neighborhood code element set, and determining a second neighborhood code element set with the largest matching number in the plurality of second neighborhood code element sets as a target second neighborhood code element set; and (Huang, page 8, [0034], “S51, construct the reference projector column number of the decoding point. The 8-nearest neighbor search method is used to search for the 8 nearest decoding points of the current decoding point Si(x,y,u,v) in the image space, denoted as U1,...,U8. These 8 neighboring decoding points have their own Candidate projector column numbers, denoted as CANDu1,...,CANDu8; merge the candidate projector column numbers of these neighboring decoding points into a sequence, denoted as UCANDi(u1,...,uw), where w is the neighboring decoding point candidate projector The sum of the number of column number elements, this sequence is called the reference projector column number of the current decoding point;” and page 9, [0042]-[0046], “A spatial coding structured light three-dimensional scanning method and system of the present invention has the following beneficial effects compared with the prior art: ... 4)Since dots are used as the basic code element, the design of the coding pattern facilitates the extraction of high-precision matching points;”)
determining a candidate code element corresponding to the target second neighborhood code element set as the target code element. (Huang, page 8, [0039], “On the basis of the above technical solutions, the preferred implementation of S6 is: traverse the image encryption points Ti(x,y,u,v) one by one, 0≤i≤t, and search for the upper, lower and left of the image encryption points according to their neighbors. , Right adjacent decoding point, record the column number values of these four adjacent decoding points as c up, c down, c left, c right; if c up = c down = c left +1 = c right -1, then The column number of the current image encryption point Ti(x,y,u,v) is c, at this time the encryption point is Ti(x,y,u,v,c), and c is the column number value of the obtained encryption point ; Otherwise, delete the current encryption point; this step continues until the judgment of all encryption points is completed.” and page 8, [0034], “S51, construct the reference projector column number of the decoding point. The 8-nearest neighbor search method is used to search for the 8 nearest decoding points of the current decoding point Si(x,y,u,v) in the image space, denoted as U1,...,U8. These 8 neighboring decoding points have their own Candidate projector column numbers, denoted as CANDu1,...,CANDu8; merge the candidate projector column numbers of these neighboring decoding points into a sequence, denoted as UCANDi(u1,...,uw), where w is the neighboring decoding point candidate projector The sum of the number of column number elements, this sequence is called the reference projector column number of the current decoding point;” and page 9, [0042]-[0046], “A spatial coding structured light three-dimensional scanning method and system of the present invention has the following beneficial effects compared with the prior art: ... 4)Since dots are used as the basic code element, the design of the coding pattern facilitates the extraction of high-precision matching points;”)
17. As per claim 3, Huang in view of Nishina, and further in view of Hansen discloses: The method as claimed in claim 2, wherein the determining the first neighborhood code element set of the any code element in the second image comprises: (See rejection for claim 2.)
determining a plurality of feature points of the any code element; and (Huang, pages 7-8, [0031], “S41: Determine the candidate projector column number of the decoding point by using the horizontal epipolar line constraint relationship. According to the calibration parameters of the projector and the camera, the sub-pixel image coordinates (u, v) of the image decoding point Si (x, y, u, v) are converted into camera epipolar image coordinates, and the row coordinate value of the epipolar image coordinates ... this process continues until the judgment of the candidate projector column number for all image decoding points is completed; if the candidate projector column number cannot be found according to the horizontal epipolar line constraint, it is directly deleted The current decoding point.” and page 7, [0029], “S35, precise extraction of encrypted points. The Gaussian ellipse fitting method is used to extract the coordinates of the entire pixel coordinates of the image encryption point to obtain the sub-pixel coordinates (u,v) of the image encryption point, then the image encryption point is recorded as Ti(x,y,u,v), 0 ≤ i ≤ t.”)
determining code elements with the nearest distances on a left side and a right side of the epipolar line where the plurality of feature points are located as neighborhood code elements of the any code element, and forming the neighborhood code elements of the any code element into the first neighborhood code element set. (Huang, page 6, [0015], “S4: Determine the candidate projector column number of the decoding point by using the relationship between the horizontal epipolar line constraint and the relative displacement of the neighborhood;” and page 7, [0031], “S41: Determine the candidate projector column number of the decoding point by using the horizontal epipolar line constraint relationship. According to the calibration parameters of the projector and the camera, the sub-pixel image coordinates (u, v) of the image decoding point Si (x, y, u, v) are converted into camera epipolar image coordinates, and the row coordinate value of the epipolar image coordinates” and page 8, [0032], “In S42, the candidate projector column number of the decoding point is selected by using the relative displacement relationship of the neighborhood. For the i-th decoding point Si(x,y,u,v), search for the left and right neighborhood decoding points, and obtain the corresponding D value according to the relative displacement relationship of the symbol neighborhood, which is marked as di here;”)
18. As per claim 4, Huang in view of Nishina, and further in view of Hansen discloses The method as claimed in claim 2, wherein the determining the number of matching neighborhood code elements in the plurality of the second neighborhood code element sets with neighborhood code elements in the first neighborhood code element set comprises: (See rejection for claim 2.)
numbering all the code elements in the first image and the second image according to pixel coordinates of the feature points of the code elements; (Huang, page 7, [0028], “S34, the encrypted points are roughly extracted. ... assuming that the number of detected image encryption points is t, the integral pixel coordinates Ti(x,y) of the recorded image encryption point, 0 ≤ i ≤ t, where (x,y) is the integral pixel coordinate value.” and page 7, [0031], “S41: Determine the candidate projector column number of the decoding point by using the horizontal epipolar line constraint relationship. According to the calibration parameters of the projector and the camera, the sub-pixel image coordinates (u, v) of the image decoding point Si (x, y, u, v) are converted into camera epipolar image coordinates, and the row coordinate value of the epipolar image coordinates”)
comparing the neighborhood code elements in the first neighborhood code element set with neighborhood code elements in corresponding areas in the plurality of second neighborhood code element sets sequentially, and under a condition that code element numbers are the same, determining the neighborhood code elements to be matched with the neighborhood code elements in the corresponding areas; and (Huang, page 8, [0032], “In S42, the candidate projector column number of the decoding point is selected by using the relative displacement relationship of the neighborhood. For the i-th decoding point Si(x,y,u,v), search for the left and right neighborhood decoding points, and obtain the corresponding D value according to the relative displacement relationship of the symbol neighborhood, which is marked as di here; Candidate projector column number CANDi(c1,...,ck) of i decoding points, get the corresponding D value according to the candidate projector column number ck, here is recorded as Dk, when di=Dk, keep this candidate projection Column number, otherwise, delete; remember that the candidate projector column number of the i-th decoding point selected through the neighborhood relative displacement relationship is CANDi(c1,...,cn), where n is the selected candidate projector The number of column numbers, (c1,...,cn) is the sequence of column number values of candidate projectors; the above process continues until the deletion of candidate projector column numbers for all image decoding points is completed; if the i-th decoding point If the D value cannot be obtained, do not do this step, and directly keep all possible candidate projector column numbers.” and page 9, [0042]-[0046], “A spatial coding structured light three-dimensional scanning method and system of the present invention has the following beneficial effects compared with the prior art: ... 4)Since dots are used as the basic code element, the design of the coding pattern facilitates the extraction of high-precision matching points;”)
determining the number of matching each of the plurality of second neighborhood code element sets with the neighborhood code elements in the first neighborhood code element set. (Huang, page 8, [0034], “S51, construct the reference projector column number of the decoding point. The 8-nearest neighbor search method is used to search for the 8 nearest decoding points of the current decoding point Si(x,y,u,v) in the image space, denoted as U1,...,U8. These 8 neighboring decoding points have their own Candidate projector column numbers, denoted as CANDu1,...,CANDu8; merge the candidate projector column numbers of these neighboring decoding points into a sequence, denoted as UCANDi(u1,...,uw), where w is the neighboring decoding point candidate projector The sum of the number of column number elements, this sequence is called the reference projector column number of the current decoding point;” and page 9, [0042]-[0046], “A spatial coding structured light three-dimensional scanning method and system of the present invention has the following beneficial effects compared with the prior art: ... 4)Since dots are used as the basic code element, the design of the coding pattern facilitates the extraction of high-precision matching points;”)
19. As per claim 5, Huang in view of Nishina, and further in view of Hansen discloses: The method as claimed in claim 1, wherein prior to determining, in the first image, the plurality of candidate code elements corresponding to the any code element in the second image, the method further comprises: (See rejection for claim 1.)
adjusting the first image and the second image to a plane in a same direction. (Nishina, [0031], “If the direction in which the projection unit 110 and the camera 120 are aligned is defined as the horizontal direction (X direction), then the epipolar lines in the projection pattern will be in the horizontal direction.”; Examiner’s note: The first image would be associated with the projection unit and the second image would be associated with the camera. Thus, when they are adjusted to align in “the horizontal direction (X direction), then the epipolar lines in the projection pattern will be in the horizontal direction.”)
20. Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to modify the method of claim 1 of Huang in view of Hansen to include the disclosure of adjusting the first image and the second image to a plane in a same direction, of Nishina. The motivation for this modification could have been to ensure that code element being used is able to more easily match from the first image to the second image. This will help improve accuracy of the overall reconstruction.
21. As per claim 6, Huang in view of Nishina, and further in view of Hansen discloses: The method as claimed in claim 1, wherein the method further comprises: (See rejection for claim 1.)
selecting at least one code element from a preset code element set, wherein the code element in the preset code element set at least comprises a line segment with a preset length and two endpoints corresponding to the line segment; and (Nishina, [0015]-[0016], “More specifically, the lighting device according to this embodiment is a lighting device that projects patterned light having a predetermined pattern, wherein the predetermined pattern is a pattern obtained by cutting multiple parallel lines in multiple rows at random positions and intervals to divide them into multiple line segments, and the perpendicular lines of the line segments passing through the upper and lower ends of each line segment intersect with line segments of adjacent rows on either side of the line segment. In this embodiment as well, it is preferable that the perpendicular line of a line segment passing through the upper end position of the line segment in the pattern intersects the line segment at a position at least a predetermined length above the lower end position of the line segment in the adjacent row on either side of the line segment, and the perpendicular line of a line segment passing through the lower end position of the line segment intersects the line segment at a position at least a predetermined length below the upper end position of the line segment in the adjacent row on either side of the line segment.”)
randomly distributing the plurality of at least one code element in the preset direction and at a preset distance to form the first image. (Nishina, [0003], “A line pattern with random cuts is a pattern in which multiple parallel lines (line segments) are cut at random positions and intervals, dividing them into multiple line segments. For simplicity, in this specification, a line pattern having random cuts will be simply referred to as a "line pattern." If the epipolar line is horizontal (horizontal direction, X direction), then each line in the line pattern is generally considered to be parallel to the vertical direction (vertical direction, Y direction). In line patterns, encoding is performed using the relationship between the relative positions (coordinates) of a given line segment and the surrounding line segments' Y coordinates.” and [0059], “Furthermore, since the overlap of adjacent line segments in the Y direction is set to a predetermined distance or greater, the possibility of misdetection of adjacent line segments can be suppressed even if the calibration is not perfect and deformation occurs, or if noise is added to the captured image.”)
22. Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to modify the method of claim 1 of Huang in view of Hansen to include the disclosure of selecting at least one code element from a preset code element set, wherein the code element in the preset code element set at least comprises a line segment with a preset length and two endpoints corresponding to the line segment and randomly distributing the plurality of at least one code element in the preset direction and at a preset distance to form the first image, of Nishina. The motivation for this modification could have been to utilize a preset code element that, when projected onto an object, can be mapped to the second image. The random distribution helps to prevent ambiguity of the mapping process and ensures that the extracted points are separate and can fully realize the 3D reconstruction.
23. As per claim 7, Huang in view of Nishina, and further in view of Hansen discloses: The method as claimed in claim 1, wherein after the determining the target code element set corresponding to each code element in the second image, the method further comprises: (See rejection for claim 1.)
determining central lines of the code elements in the first image and the second image; and (Nishina, [0044], “The line segment extraction unit 134 determines the centroid position in the luminance profile on a sub-pixel basis. After determining the centroid position in the X direction for each Y coordinate within the line segment, the line segment extraction unit 134 further smooths the line formed by the centroid positions in the X direction of multiple Y coordinates. The line segment extraction unit 134 uses the centroid position of the sub-pixel accuracy obtained in this way as the X coordinate of the line segment.”)
determining a mapping relationship between the central lines of the code elements in the first image and the second image according to a mapping relationship among the target code elements corresponding to all the code elements in the second image. (Nishina, [0046]-[0047], “Through the above process, the position (coordinates) of line segments in the captured image can be determined at the sub-pixel level. Once the line segment extraction process S302 is completed, in step S303 (Figure 3), the correspondence processing unit 135 determines the correspondence between the line segments in the captured image and the line segments in the pattern. Figure 6 shows the detailed flow of the mapping process S303. The mapping process S303 identifies six adjacent line segments for each line segment in the captured image during the processing of loop L1.” and [0044], “The line segment extraction unit 134 determines the centroid position in the luminance profile on a sub-pixel basis. After determining the centroid position in the X direction for each Y coordinate within the line segment, the line segment extraction unit 134 further smooths the line formed by the centroid positions in the X direction of multiple Y coordinates. The line segment extraction unit 134 uses the centroid position of the sub-pixel accuracy obtained in this way as the X coordinate of the line segment.”)
25. Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to modify the method of claim 1 of Huang in view of Hansen to include the disclosure of determining central lines of the code elements in the first image and the second image and determining a mapping relationship between the central lines of the code elements in the first image and the second image according to a mapping relationship among the target code elements corresponding to all the code elements in the second image, of Nishina. The motivation for this modification could have been to utilize the central lines and mapping relationship to match the code elements from the first image to the second image. This will help build a correspondence between the two images for 3D point extraction.
25. Claim 9 is similar in scope to independent claim 1 except for different limitations that Huang in view of Nishina, and further in view of Hansen discloses: A system for three-dimensional reconstruction, comprising: (Huang, page 5, [0001], “The invention relates to the field of three-dimensional measurement, in particular to a three-dimensional scanning method and system for spatially coded structured light.”)
an image acquisition device, a projection device, and a first processor, wherein the projection device is configured to project a first image onto a surface of a measured object; (Huang. page 6, [0013]-[0014], “S2: Project the pre-encoded single-frame spatial encoding pattern onto the surface of the object to be measured through a projector; S3, the camera shoots the spatial coding pattern modulated by the object, and extracts the image decoding points and encryption points;” and page 13, [0093], “In addition, the present invention is not only applicable to a single-camera structured light three-dimensional measurement system, but also applicable to a multi-camera structured light three-dimensional measurement system. In specific implementation, the above process can be automatically run by computer software, and the system device that runs this method should also be within the scope of protection.”)
26. Claim 10, which is similar in scope to independent claims 1 and 9, is thus rejected under the same rationale as described above. The motivation for this modification is the same as claim 1.
27. As per claim 13, Huang in view of Nishina, and further in view of Hansen discloses: The method as claimed in claim 1, wherein the second image is an image acquired by the image acquisition device after the first image is projected onto the surface of the measured object (Huang. page 6, [0013]-[0014], “S2: Project the pre-encoded single-frame spatial encoding pattern onto the surface of the object to be measured through a projector; S3, the camera shoots the spatial coding pattern modulated by the object, and extracts the image decoding points and encryption points;”) and deformed, (Nishina, [0041], “The pattern light, consisting of straight lines, is captured as a distorted shape according to the shape of the object being measured, 150.”) the plurality of code elements in the first image have the same orientation, (Nishina, [0041], “The pattern light, consisting of straight lines, is captured as a distorted shape according to the shape of the object being measured, 150. Here, each point in the projection light pattern lies on the epipolar line in the captured image. Furthermore, in this embodiment, it is assumed that the epipolar line is oriented horizontally in the captured image. Therefore, the Y-coordinate position (vertical position) of each point in the projection light pattern does not change in the captured image.”) and the spacing between the code elements is determined randomly. (Nishina, [0076], “Furthermore, while the above explanation describes random generation and verification of each column, verification could also be performed each time a line segment or gap (vertical spacing) is randomly generated.”)
28. Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to modify the method of claim 1 of Huang in view of Hansen to include the disclosure of the first image deformed on an object to be measured and the plurality of code elements in the first image have the same orientation, and the spacing between the code elements is determined randomly, of Nishina. The motivation for this modification could have been to help ensure that the code elements from the first image and the second image can build a correspondence between the two images for 3D point extraction. Also, the random spacing between code elements will help remove any ambiguity in mapping the first image to the second image.
29. As per claim 15, Huang in view of Nishina, and further in view of Hansen discloses: The method as claimed in claim 4, wherein the code element corresponds to the code element number one by one; (See Hansen, page 46, lines 28-30 below.)
the method further comprises:
determining the code element number by the positions of the code elements stored in a database; or, (Hansen, page 46, lines 28-30, “Such reference structured light database comprises a plurality of sets of reference structured light data sets each linked to a unique code may e.g. form part of the depiction system.”)
determining the code element number of each code element in an order in which the code elements are extracted from the database. (Hansen, page 46, lines 23-28, “The reference structured light data set may e.g. be transmitted to the computer system by an operator and/or the computer system may be configured for acquiring the reference structured light data set from a database e.g. in response to an instruction from an operator and/or based on an code included in the structured light beam, such as an optically detectable code.”)
30. Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to modify the method of claim 4 of Huang in view of Nishina to include the disclosure of determining the code element number by the positions of the code elements stored in a database or determining the code element number of each code element in an order in which the code elements are extracted from the database, of Hansen. The motivation for this modification could have been to keep track of which code element properly maps to the second image so that it can be used for 3D reconstruction.
31. Claim 16, which is similar in scope to independent claims 1 and 9, is thus rejected under the same rationale as described above. The motivation for this modification is the same as claim 1.
32. Claim 17, which is similar in scope to dependent claim 2 and independent claim 9, is thus rejected under the same rationale as described above. The motivation for this modification is the same as claim 2.
33. Claim 19, which is similar in scope to dependent claims 4 and 17 and independent claim 9, is thus rejected under the same rationale as described above. The motivation for this modification is the same as claim 4.
34. Claim 20, which is similar in scope to dependent claim 5 and independent claim 9, is thus rejected under the same rationale as described above. The motivation for this modification is the same as claim 5.
35. Claim 21, which is similar in scope to dependent claim 6 and independent claim 9, is thus rejected under the same rationale as described above. The motivation for this modification is the same as claim 6.
36. Claim 22, which is similar in scope to dependent claim 7 and independent claim 9, is thus rejected under the same rationale as described above. The motivation for this modification is the same as claim 7.
37. Claim 23, which is similar in scope to dependent claim 13 and independent claim 9, is thus rejected under the same rationale as described above. The motivation for this modification is the same as claim 13.
38. Claim 14 is rejected under 35 U.S.C. 103 as being unpatentable over Huang et al. (CN-111336949-A, hereinafter "Huang") in view of Nishina et al. (JP-2017/053812-A, hereinafter "Huang"), further in view of Hansen et al. (WO-2018/171851-A1, hereinafter "Hansen"), and further in view of Song et al. (CN-108122254-A, hereinafter "Song").
39. As per claim 14, Huang in view of Nishina, and further in view of Hansen discloses: The method as claimed in claim 1, (See rejection for claim 1.) [[wherein feature points of the code elements comprise at least two extractable gray feature point patterns, and the extractable gray features comprise any one of the following:]]
[[two endpoints of a line segment, a graphic corner point, an intersection point, a center point.]]
40. Huang in view of Nishina, and further in view of Hansen doesn't explicitly disclose but Song discloses: [[The method as claimed in claim 1,]] wherein feature points of the code elements comprise at least two extractable gray feature point patterns, and the extractable gray features comprise any one of the following: (See Song, [0018] below.)
two endpoints of a line segment, a graphic corner point, an intersection point, a center point.
(Song, [0018], “The feature point calculation unit is used to locate the initial position of the graphic feature points in each of the coded element images using a preset corner detection algorithm, and to calculate the auxiliary coded feature points of the coded element images based on the initial position and the gray value of the coded element images.”)
41. Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to modify the method of claim 1 of Huang in view of Nishina, and further in view of Hansen to include the disclosure of code elements comprise at least two extractable gray feature point patterns, and the extractable gray features comprise of any of the following: two endpoints of a line segment, a graphic corner point, an intersection point, a center point, of Song. The motivation for this modification could have been to utilize the feature points extracted from the code elements to distinguish different parts of the object being projected onto. These features will help to define the 3D reconstruction of the object.
Conclusion
42. Any inquiry concerning this communication or earlier communications from the examiner should be directed to MATTHEW CLOTHIER whose telephone number is (571)272-4667. The examiner can normally be reached Mon-Fri 8:00am-4:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kent Chang can be reached at (571)272-7667. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MATTHEW CLOTHIER/Examiner, Art Unit 2614
/KENT W CHANG/Supervisory Patent Examiner, Art Unit 2614
