DETAILED ACTION
Response to Arguments
The amendments filed 10/15/2025 have been entered and made of record.
Applicant's arguments filed 10/15/2025 have been fully considered, however Applicant’s arguments are not persuasive:
A. Claims 1-2, an 8-14 Rejection Under 35 U.S.C. § 103 as being unpatentable over DOEMLING as modified by AFROUZI;
With regarding to claim 1, Applicant states (in page 11 of the Remarks of 10/15/2025), DOEMLING fails to disclose at least limitations (b)-(f), and specifically, DOEMLING is silent regarding frequences or counts of points, as limited in limitation (b):
(b) determining, for each of the variables, a respective frequency distribution in relation to the values of this variable determined for the different points in each case;
However, the Examiner disagrees, because:
as demonstrated in DOEMLING’s Fig. 3 (c), and Fig. 4 as reproduced below:
PNG
media_image1.png
917
802
media_image1.png
Greyscale
PNG
media_image2.png
711
1131
media_image2.png
Greyscale
In Fig. 4, for Frame 1, and Frame 2 respectively, Generate “SGM”s; which is disclosed in
“[0041] A Gaussian Mixture Model of a geometric primitive may comprise one or more Gaussian distributions, with each Gaussian distribution having a specific mean, a covariance and a composition weight.”
Apparently above “each Gaussian distribution having a specific mean, a covariance and a composition weight.” read on claimed limitation of “a respective frequency distribution in relation to the values of this variable determined for the different points in each case”, because DOEMLING’s “each Gaussian distribution having a specific mean, a covariance and a composition weight.” including frequences or counting the number of points in point clusters/groups, which represent objects, as demonstrated in Fig. 3 (c), and Fig. 4; and, further as shown in DOEMLING’s [0044]:
“Moreover, the covariance of each Gaussian distribution can be determined based on the geometric parameters of the corresponding black cell. Additionally, the composition weights of Gaussian distributions may be determined by the area ratios between corresponding black cells.”
Again, “the composition weights of Gaussian distributions may be determined by the area ratios between corresponding black cells.” dead on a claimed limitation of “a respective frequency distribution in relation to the values of this variable determined for the different points in each case”;
It is pointed out that claims do not specifically limit the characteristic of “variable”, neither limit the characteristic of “the values of this variable determined for the different points in each case”; so that it is proper to have DOEMLING’s disclosures of “calculating the Gaussian Mixture Model” including “Gaussian distributions” as defined in
[0044] Alternatively, when calculating the Gaussian Mixture Model of a geometric primitive belonging to planar or cube primitives, the boundary of the geometric primitive may be used to generate a quadtree or octree partition of the geometric primitive. Then a Gaussian distribution may be put into each blank cell which is located within the geometric primitive, thereby forming the Gaussian Mixture Model of the geometric primitive. The number of partitions, that is, the number of Gaussian distributions within the Gaussian Mixture Model may depend on accuracy requirement. It is clear that the mean of each Gaussian distribution put into each black cell is the center of the blank cell. Moreover, the covariance of each Gaussian distribution can be determined based on the geometric parameters of the corresponding black cell.
clearly provide enough and sufficient examples for “variable” and “the values of this variable determined for the different points in each case” being applied in “calculating the Gaussian Mixture Model” including “Gaussian distributions”; also see DOEMLING: e.g., Fig. 1, and, --[0024] A point cloud is a set of data points in space. The point cloud is generally produced by a sensor, which measures a large number of points on the external surfaces of objects around it. Each point within the point cloud may have a three dimensional coordinate (x, y, z) and other information (e.g. the color information, the reflection intensity, the normal direction, etc.). The point cloud may be used for many purposes, including creating 3D maps, object recognition and object tracking. In order to achieve these purposes, the point cloud should first be segmented based on some attributes of the points within the point cloud to determine which points belong to a particular object. Due to some reasons such as the limitation of the field of view of the sensor, one frame of point cloud may not obtain the point cloud of the whole object.--, in [0024]-[0025];
Applicant also asserts (in page 11 of the Remarks of 10/15/2025) that, DOEMLING also fails to approximating frequency distributions by a linear combination
of probability density functions as limitation (c) requires:
(c) approximating each of the frequency distributions by means of a respective linear combination of a finite number of one-dimensional probability density functions assigned to the underlying variable;
However, the Examiner disagrees, because:
DOEMLING clearly discloses approximating each of the frequency distributions by a linear combination of probability density functions, in
-- a method for representing a point cluster may comprise (a) generating a Gaussian Mixture Model for the point cluster; (b) generating a signature for identifying the point cluster; and (c) generating a SGM model for representing the point cluster.
[0036] A Gaussian Mixture Model is a type of density model which comprises a number of Gaussian distributions. For example, a typical Gaussian Mixture Model in one dimension may take the form of formula (1) as below:
p(x)=Σ.sub.k=1.sup.Kπ.sub.kp(x|k) (1)
wherein K is the number of Gaussian distributions contained in the Gaussian Mixture Model, π.sub.k is the composition weight of the k.sup.th Gaussian distribution, and p(x|k) is the probability density function of the k.sup.th Gaussian distribution which is defined by its average μ.sub.k and covariance b.sub.k. In particular, a typical Gaussian distribution (i.e. the probability density function of the Gaussian distribution) in one dimension may take the form of formula (2)--, in [0035]-[0039];
such as in above Equation (1), “Σ” apparently is a linear combination of “p(x|k) is the probability density function of the k.sup.th Gaussian distribution”.
For the segmenting the image, with the steps (d)-(g);
It is pointed out that Applicant’s discussions and explanations of the segmenting processes in view of Fig. 2, (as reproduced in page 9 of the Remarks of 10/15/2025); however, the claimed limitations of the steps (c)-(f) do not match with, or be consistent with the processes in Fig. 2: at least, the claimed limitations of the steps (c)-(f) miss the critical step, or the component of “The intersection of the probability density functions f₁(z) and f₂(z) are then determined as illustrated in view 235 by the black, horizontal dashed line.”; it is therefore not specifically limited in the claimed limitations of the steps (c)-(f), how to segment O1, and O2 without the separation line of “The intersection of the probability density functions f₁(z) and f₂(z) are then determined as illustrated in view 235 by the black, horizontal dashed line.”;
It is further pointed out that claim does not specifically define what, and how to derive “product of m probability density functions”;
Nevertheless, Applicant asserts that DOEMLING as modified by AFROUZI do not disclose the steps (d)-(g) for segmenting the image,
However, the Examiner disagrees, because:
DOEMLING discloses (d) segmenting the image so that (see DOEMLING: e.g., --[0040] In an example, the Gaussian Mixture Model for the point cluster may be generated by using Gaussian Mixture Models of geometric primitives to approximate a shape of the point cluster. It is noted that the shape of the point cluster may be determined by using the geometric information of the point cluster or other information of the point cluster. The Gaussian Mixture Model for the point cluster may be generated by using Gaussian Mixture Models for geometric primitives alone or in combination. As an example, a point cluster has a shape approximating that of a planar ellipse primitive. This shape may be generated by using the Gaussian Mixture Model for the planar ellipse primitive based on geometric parameters of the point cluster. As another example, a point cluster has a shape approximating that of a triangle primitive and a line segment primitive. This shape may be generated by using the combination of a Gaussian Mixture Model for the triangle primitive and a Gaussian Mixture Model for the line segment primitive based on geometric parameters of the point cluster.
[0041] A Gaussian Mixture Model of a geometric primitive may comprise one or more Gaussian distributions, with each Gaussian distribution having a specific mean, a covariance and a composition weight. The number of Gaussian distributions in the Gaussian Mixture Model may be selected based on accuracy requirement. The Gaussian Mixture Models for geometric primitives may be expressed by the geometric parameters of the geometric primitives.
[0042] The Gaussian Mixture Models for point clusters may be generated for example via a substitution method, i.e., by substituting geometric parameters of geometric primitives for concrete geometric parameters of the point clusters. In the case that a point cluster has a shape approximating that of several geometric primitives in combination, generating the Gaussian Mixture Model for the point cluster may also comprise combining Gaussian Mixture Models for the geometric primitives to generate the Gaussian Mixture Model for the point cluster.
[0043] In an example, calculating a Gaussian Mixture Model of a geometric primitive may comprises sampling the geometric primitive and then using data points resulting from the sampling to determine parameters of each Gaussian distribution, such as the specific mean, covariance and composition weight thereof. For example, the specific mean, covariance and composition weight of the Gaussian distribution may be calculated for example by using Expectation Maximization (EM) Algorithm.-- , in [0040]-[0043]; also see:
[0031] The mapped point clusters are denoted as 110′, 112′ and 114′ in FIG. 2. Points within the dashed circles 210, 220 and 230 are the real points obtained in Frame 2. As shown in FIG. 2, all points within the dashed circle 210 in Frame 2 overlap with the mapped point cluster 110′. Therefore, there is no need to perform segmentation for all points within the dashed circle 210. Four points within the dashed circle 220 overlap with the mapped point cluster 112′. Therefore, only other five points within the dashed circle 220 need to be segmented. No points within the dashed circle 230 overlaps with any mapped point cluster and thus all points within the dashed circle 230 need to be segmented. Totally, only 7 out of 20 points in Frame 2 need to be segmented, which may lead to substantial resource-saving.
[0032] After these seven points in Frame 2 have been segmented, there may be five point clusters 110′, 112′, 240, 114′ and 230. It can then be determined whether these point clusters can be merged according to certain rules. For example, since the point clusters 114′ and 230 are close to each other and their points have similar flatness, these two point clusters may be merged into one point cluster 260. As another example, the point clusters 112′ and 240 are close to each other and their points have similar reflection rate. Therefore, these two point clusters may be combined into one point cluster 250. Finally, there are totally three point clusters 110′, 250 and 260 which represent the segmentation result of Frame 2.--, in [0031]-[0032]);
Apparently, as discussed above, and demonstrated by DOEMLING’s Fig. Fig. 3 (c), and Fig. 4; DOEMLING’s calculating a Gaussian Mixture Model of a geometric primitive including the probability density functions to segment the image to identify/separate objects as point clusters based on to determine parameters of each Gaussian distribution, such as the specific mean, covariance and composition weight thereof. For example, the specific mean, covariance and composition weight of the Gaussian distribution may be calculated for example by using Expectation Maximization (EM) Algorithm.
AFROUZI is cited and applied in the Office Action to particularly cover the claim limitation element of “product of m probability density functions”, in:
AFROUZI teaches (d) segmenting the image so that:
(e) in the case m = 1 each of the probability density functions and in the case m > 1 each product of m probability density functions, where one of the assigned probability density functions per variable is represented in the product in each case, is uniquely assigned a respective segment of the image;
(f) respectively assigning each point of the point cloud to that segment whose assigned probability density function in the case of m = 1 or whose assigned product in the case of m > 1 has the relatively largest function value or product value among the probability density functions or products at the location which is determined by the values of the m variables assigned to the point (see EBRAHIMI AFROUZI: e.g., -- the processor of the robot may perform segmentation wherein an object captured in an image is separated from other objects and the background of the image.--, in [0936], and, --the processor may convert the a priori probability P(ω.sub.j) to an a posteriori probability P(x), i.e., the probability of a state of the object being of given the feature value x has been observed. P(ω.sub.j) is the probability of observing the feature value x given the state of the object is ω.sub.j. The product of P(ω.sub.j)P(ω.sub.j) is a significant factor in determining the a posteriori probability whereas the evidence P(x) is a normalizer to ensure the a posteriori probabilities sum to one. In some embodiments, the processor considers more than one feature by replacing the scalar feature value x by a feature vector x, wherein x is of a multi-dimensional Euclidean space R.sup.n or otherwise the feature space. For the feature vector x, a n-component vector-valued random variable, P(ω.sub.j) is the state-conditional probability density function for x, i.e., the probability density function for x conditioned on ω.sub.j being the true category.-- ,in [01071]-[0174]);
DOEMLING and EBRAHIMI AFROUZI are combinable as they are in the same field of endeavor: segmenting the image, and segmenting objects in the image. Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to further modify DOEMLING’s method using EBRAHIMI AFROUZI’s teachings by including (d) segmenting the image so that: (e) in the case m = 1 each of the probability density functions and in the case m > 1 each product of m probability density functions, where one of the assigned probability density functions per variable is represented in the product in each case, is uniquely assigned a respective segment of the image;
(f) respectively assigning each point of the point cloud to that segment whose assigned probability density function in the case of m = 1 or whose assigned product in the case of m > 1 has the relatively largest function value or product value among the probability density functions or products at the location which is determined by the values of the m variables assigned to the point to DOEMLING’s generating the Gaussian Mixture Model for the point clusters in order to perform segmentation wherein an object captured in an image is separated from other objects and the background of the image (see EBRAHIMI AFROUZI: e.g. in [0936], and [1071]-[1074]);
DOEMLING as modified by EBRAHIMI AFROUZI further disclose identifying at least one segment of those which were each assigned at least a predetermined minimum number of points as a representative of a respective recognized object (see DOEMLING: e.g., --[0040] In an example, the Gaussian Mixture Model for the point cluster may be generated by using Gaussian Mixture Models of geometric primitives to approximate a shape of the point cluster. It is noted that the shape of the point cluster may be determined by using the geometric information of the point cluster or other information of the point cluster. The Gaussian Mixture Model for the point cluster may be generated by using Gaussian Mixture Models for geometric primitives alone or in combination. As an example, a point cluster has a shape approximating that of a planar ellipse primitive. This shape may be generated by using the Gaussian Mixture Model for the planar ellipse primitive based on geometric parameters of the point cluster. As another example, a point cluster has a shape approximating that of a triangle primitive and a line segment primitive. This shape may be generated by using the combination of a Gaussian Mixture Model for the triangle primitive and a Gaussian Mixture Model for the line segment primitive based on geometric parameters of the point cluster.
[0041] A Gaussian Mixture Model of a geometric primitive may comprise one or more Gaussian distributions, with each Gaussian distribution having a specific mean, a covariance and a composition weight. The number of Gaussian distributions in the Gaussian Mixture Model may be selected based on accuracy requirement. The Gaussian Mixture Models for geometric primitives may be expressed by the geometric parameters of the geometric primitives.
[0042] The Gaussian Mixture Models for point clusters may be generated for example via a substitution method, i.e., by substituting geometric parameters of geometric primitives for concrete geometric parameters of the point clusters. In the case that a point cluster has a shape approximating that of several geometric primitives in combination, generating the Gaussian Mixture Model for the point cluster may also comprise combining Gaussian Mixture Models for the geometric primitives to generate the Gaussian Mixture Model for the point cluster.
[0043] In an example, calculating a Gaussian Mixture Model of a geometric primitive may comprises sampling the geometric primitive and then using data points resulting from the sampling to determine parameters of each Gaussian distribution, such as the specific mean, covariance and composition weight thereof. For example, the specific mean, covariance and composition weight of the Gaussian distribution may be calculated for example by using Expectation Maximization (EM) Algorithm.-- , in [0040]-[0043]; also see EBRAHIMI AFROUZI: e.g., -- the processor of the robot may perform segmentation wherein an object captured in an image is separated from other objects and the background of the image.--, in [0936], and, --the processor may convert the a priori probability P(ω.sub.j) to an a posteriori probability P(x), i.e., the probability of a state of the object being of given the feature value x has been observed. P(ω.sub.j) is the probability of observing the feature value x given the state of the object is ω.sub.j. The product of P(ω.sub.j)P(ω.sub.j) is a significant factor in determining the a posteriori probability whereas the evidence P(x) is a normalizer to ensure the a posteriori probabilities sum to one. In some embodiments, the processor considers more than one feature by replacing the scalar feature value x by a feature vector x, wherein x is of a multi-dimensional Euclidean space R.sup.n or otherwise the feature space. For the feature vector x, a n-component vector-valued random variable, P(ω.sub.j) is the state-conditional probability density function for x, i.e., the probability density function for x conditioned on ω.sub.j being the true category.-- ,in [01071]-[0174]).
B. Claims 3-7 rejected under 35 U.S.C. 103 as being patentable over DOEMLING as modified by EBRAHIMI AFROUZI, and ECKART.
Applicant did not raise additional arguments other that repeating the arguments used in for claim 1: However, the Examiner has fully addressed that DOEMLING’s generating a Gaussian Mixture Model for the point cluster, which has been discussed above for claim 1, including “For example, a typical Gaussian Mixture Model in one dimension may take the form of formula (1) as below:
p(x)=Σ.sub.k=1.sup.Kπ.sub.kp(x|k) (1)
and, such as in above Equation (1), “Σ” apparently is a linear combination of “p(x|k) is the probability density function of the k.sup.th Gaussian distribution”.
As disclosed in detail in DOEMLING’s [0031]-[0032], and [0035]-[0043], and further demonstrated in Fig. 3 , and Fig. 4;
exactly read on claimed limitation of (b) determining, for each of the variables, a respective frequency distribution in relation to the values of this variable determined for the different points in each case.
Therefore, claims 1-14 are still not patentably distinguishable over the prior art reference(s). Further discussions are addressed in the prior art rejection section below.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 8-14 are rejected under 35 U.S.C. 103 as being patentable over DOEMLING (US 20210248752 A1, Date Filed: 2018-06-15), and in view of EBRAHIMI AFROUZI (US 20220187841 A1, claims priority of US-Provisional-application US 63124004, 20201210, the cited disclosures in this Office Action have been found consistent with the corresponding contents in the US-Provisional-application US 63124004, such as in page 18, page 22, and pages 27-28).
Re Claim 1, DOEMLING discloses a method for recognizing one or more objects represented in an image on the basis of an M-dimensional point cloud of a plurality n of points, where M > 1 (see DOEMLING: e.g., -- 3D scanners (e.g. lidar, stereo-camera, time-of-flight camera, etc.) are now widely used in automatic driving systems. Such 3D scanners measure a large number of points on an object's surface, and often output a point cloud as a data file. The point cloud represents the set of points that the 3D scanners has measured. As is known in the art, the point cloud may be used for many purposes, including creating 3D maps, object recognition and object tracking. In order to achieve these purposes, point cloud should first be segmented based on some attributes of the points within the point cloud to obtain clusters of point cloud. The resulting clusters of point cloud can then be described using a suitable model so as to represent a particular object.--, in [0002], and, Fig. 1, and, --[0024] A point cloud is a set of data points in space. The point cloud is generally produced by a sensor, which measures a large number of points on the external surfaces of objects around it. Each point within the point cloud may have a three dimensional coordinate (x, y, z) and other information (e.g. the color information, the reflection intensity, the normal direction, etc.). The point cloud may be used for many purposes, including creating 3D maps, object recognition and object tracking. In order to achieve these purposes, the point cloud should first be segmented based on some attributes of the points within the point cloud to determine which points belong to a particular object. Due to some reasons such as the limitation of the field of view of the sensor, one frame of point cloud may not obtain the point cloud of the whole object.--, in [0024]-[0025]), the method comprising:
determining, for each of a number m of specific one-dimensional variables, where m >0, a respective assigned value of the variable for each of the points on the basis of its position or characteristics (see DOEMLING: e.g., Fig. 1, and, --[0024] A point cloud is a set of data points in space. The point cloud is generally produced by a sensor, which measures a large number of points on the external surfaces of objects around it. Each point within the point cloud may have a three dimensional coordinate (x, y, z) and other information (e.g. the color information, the reflection intensity, the normal direction, etc.). The point cloud may be used for many purposes, including creating 3D maps, object recognition and object tracking. In order to achieve these purposes, the point cloud should first be segmented based on some attributes of the points within the point cloud to determine which points belong to a particular object. Due to some reasons such as the limitation of the field of view of the sensor, one frame of point cloud may not obtain the point cloud of the whole object.--, in [0024]-[0025]);
determining, for each of the variables, a respective frequency distribution in relation to the values of this variable determined for the different points in each case (see DOEMLING: e.g., --[0026] As shown on the left side of FIG. 1, the first frame of point cloud “Frame 1” may be obtained by the sensor at time t.sub.1. In Frame 1, only 24 points are depicted for ease of illustration. It should be noted that a frame of point cloud may contain hundreds or thousands of points. In order to segment Frame 1, various well-known segmentation algorithms may be employed. These segmentation algorithms may include the K-means algorithm, the K-NearestNeighbor algorithm, the GMM algorithm and/or the region growing algorithm. After the segmentation, Frame 1 may be segmented into three point clusters which are shown within the dashed circles 110, 112 and 114 in FIG. 1. The first point cluster 110 may include 18 points. The second point cluster 112 may include 4 points. The third pint cluster 114 may include 2 points. Each point cluster may have attributes such as length, width, thickness, orientation, or the ratio of main axes, and may indicate that points within the point cluster belong to the same object.
[0027] At time t.sub.2, the sensor may, for example, have been rotated for certain degree in the X-Y plane in the Cartesian coordinate system since t.sub.1 and may obtain the second frame of point cloud “Frame 2”. Frame 2 may also be segmented using abovementioned segmentation algorithms and may be segmented into three point clusters 120, 122 and 124. The fourth point cluster 120 may include 9 points. The fifth point cluster 122 may include 9 points. The sixth point cluster 124 may include 2 points.--, in [0026]-[0027], and, -- a method for representing a point cluster may comprise (a) generating a Gaussian Mixture Model for the point cluster; (b) generating a signature for identifying the point cluster; and (c) generating a SGM model for representing the point cluster.
[0036] A Gaussian Mixture Model is a type of density model which comprises a number of Gaussian distributions. For example, a typical Gaussian Mixture Model in one dimension may take the form of formula (1) as below:
p(x)=Σ.sub.k=1.sup.Kπ.sub.kp(x|k) (1)
wherein K is the number of Gaussian distributions contained in the Gaussian Mixture Model, π.sub.k is the composition weight of the k.sup.th Gaussian distribution, and p(x|k) is the probability density function of the k.sup.th Gaussian distribution which is defined by its average μ.sub.k and covariance b.sub.k. In particular, a typical Gaussian distribution (i.e. the probability density function of the Gaussian distribution) in one dimension may take the form of formula (2)--, in [0035]-[0039]);
approximating each of the frequency distributions by means of a respective linear combination of a finite number of one-dimensional probability density functions assigned to the underlying variable (see DOEMLING: e.g., --[0026] As shown on the left side of FIG. 1, the first frame of point cloud “Frame 1” may be obtained by the sensor at time t.sub.1. In Frame 1, only 24 points are depicted for ease of illustration. It should be noted that a frame of point cloud may contain hundreds or thousands of points. In order to segment Frame 1, various well-known segmentation algorithms may be employed. These segmentation algorithms may include the K-means algorithm, the K-NearestNeighbor algorithm, the GMM algorithm and/or the region growing algorithm. After the segmentation, Frame 1 may be segmented into three point clusters which are shown within the dashed circles 110, 112 and 114 in FIG. 1. The first point cluster 110 may include 18 points. The second point cluster 112 may include 4 points. The third pint cluster 114 may include 2 points. Each point cluster may have attributes such as length, width, thickness, orientation, or the ratio of main axes, and may indicate that points within the point cluster belong to the same object.
[0027] At time t.sub.2, the sensor may, for example, have been rotated for certain degree in the X-Y plane in the Cartesian coordinate system since t.sub.1 and may obtain the second frame of point cloud “Frame 2”. Frame 2 may also be segmented using abovementioned segmentation algorithms and may be segmented into three point clusters 120, 122 and 124. The fourth point cluster 120 may include 9 points. The fifth point cluster 122 may include 9 points. The sixth point cluster 124 may include 2 points.--, in [0026]-[0027], and, -- a method for representing a point cluster may comprise (a) generating a Gaussian Mixture Model for the point cluster; (b) generating a signature for identifying the point cluster; and (c) generating a SGM model for representing the point cluster.
[0036] A Gaussian Mixture Model is a type of density model which comprises a number of Gaussian distributions. For example, a typical Gaussian Mixture Model in one dimension may take the form of formula (1) as below:
p(x)=Σ.sub.k=1.sup.Kπ.sub.kp(x|k) (1)
wherein K is the number of Gaussian distributions contained in the Gaussian Mixture Model, π.sub.k is the composition weight of the k.sup.th Gaussian distribution, and p(x|k) is the probability density function of the k.sup.th Gaussian distribution which is defined by its average μ.sub.k and covariance b.sub.k. In particular, a typical Gaussian distribution (i.e. the probability density function of the Gaussian distribution) in one dimension may take the form of formula (2)--, in [0035]-[0039]);
segmenting the image so that (see DOEMLING: e.g., --[0040] In an example, the Gaussian Mixture Model for the point cluster may be generated by using Gaussian Mixture Models of geometric primitives to approximate a shape of the point cluster. It is noted that the shape of the point cluster may be determined by using the geometric information of the point cluster or other information of the point cluster. The Gaussian Mixture Model for the point cluster may be generated by using Gaussian Mixture Models for geometric primitives alone or in combination. As an example, a point cluster has a shape approximating that of a planar ellipse primitive. This shape may be generated by using the Gaussian Mixture Model for the planar ellipse primitive based on geometric parameters of the point cluster. As another example, a point cluster has a shape approximating that of a triangle primitive and a line segment primitive. This shape may be generated by using the combination of a Gaussian Mixture Model for the triangle primitive and a Gaussian Mixture Model for the line segment primitive based on geometric parameters of the point cluster.
[0041] A Gaussian Mixture Model of a geometric primitive may comprise one or more Gaussian distributions, with each Gaussian distribution having a specific mean, a covariance and a composition weight. The number of Gaussian distributions in the Gaussian Mixture Model may be selected based on accuracy requirement. The Gaussian Mixture Models for geometric primitives may be expressed by the geometric parameters of the geometric primitives.
[0042] The Gaussian Mixture Models for point clusters may be generated for example via a substitution method, i.e., by substituting geometric parameters of geometric primitives for concrete geometric parameters of the point clusters. In the case that a point cluster has a shape approximating that of several geometric primitives in combination, generating the Gaussian Mixture Model for the point cluster may also comprise combining Gaussian Mixture Models for the geometric primitives to generate the Gaussian Mixture Model for the point cluster.
[0043] In an example, calculating a Gaussian Mixture Model of a geometric primitive may comprises sampling the geometric primitive and then using data points resulting from the sampling to determine parameters of each Gaussian distribution, such as the specific mean, covariance and composition weight thereof. For example, the specific mean, covariance and composition weight of the Gaussian distribution may be calculated for example by using Expectation Maximization (EM) Algorithm.-- , in [0040]-[0043]):
Although DOEMLING discloses segmenting based on Gaussian distribution which may be calculated for example by using Expectation Maximization (EM) Algorithm,
DOEMLING however does not explicitly disclose segmenting the image so that: in the case m = 1 each of the probability density functions and in the case m > 1 each product of m probability density functions, where one of the assigned probability density functions per variable is represented in the product in each case, is uniquely assigned a respective segment of the image;
EBRAHIMI AFROUZI teaches segmenting the image so that: in the case m = 1 each of the probability density functions and in the case m > 1 each product of m probability density functions, where one of the assigned probability density functions per variable is represented in the product in each case, is uniquely assigned a respective segment of the image;
respectively assigning each point of the point cloud to that segment whose assigned probability density function in the case of m = 1 or whose assigned product in the case of m > 1 has the relatively largest function value or product value among the probability density functions or products at the location which is determined by the values of the m variables assigned to the point (see EBRAHIMI AFROUZI: e.g., -- the processor of the robot may perform segmentation wherein an object captured in an image is separated from other objects and the background of the image.--, in [0936], and, --the processor may convert the a priori probability P(ω.sub.j) to an a posteriori probability P(x), i.e., the probability of a state of the object being of given the feature value x has been observed. P(ω.sub.j) is the probability of observing the feature value x given the state of the object is ω.sub.j. The product of P(ω.sub.j)P(ω.sub.j) is a significant factor in determining the a posteriori probability whereas the evidence P(x) is a normalizer to ensure the a posteriori probabilities sum to one. In some embodiments, the processor considers more than one feature by replacing the scalar feature value x by a feature vector x, wherein x is of a multi-dimensional Euclidean space R.sup.n or otherwise the feature space. For the feature vector x, a n-component vector-valued random variable, P(ω.sub.j) is the state-conditional probability density function for x, i.e., the probability density function for x conditioned on ω.sub.j being the true category.-- ,in [01071]-[0174]);
DOEMLING and EBRAHIMI AFROUZI are combinable as they are in the same field of endeavor: segmenting the image, and segmenting objects in the image Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to further modify DOEMLING’s method using EBRAHIMI AFROUZI’s teachings by including segmenting the image so that: in the case m = 1 each of the probability density functions and in the case m > 1 each product of m probability density functions, where one of the assigned probability density functions per variable is represented in the product in each case, is uniquely assigned a respective segment of the image;
respectively assigning each point of the point cloud to that segment whose assigned probability density function in the case of m = 1 or whose assigned product in the case of m > 1 has the relatively largest function value or product value among the probability density functions or products at the location which is determined by the values of the m variables assigned to the point to DOEMLING’s generating the Gaussian Mixture Model for the point cluster in order to perform segmentation wherein an object captured in an image is separated from other objects and the background of the image (see EBRAHIMI AFROUZI: e.g. in [0936], and [1071]-[1074]);
DOEMLING as modified by EBRAHIMI AFROUZI further disclose identifying at least one segment of those which were each assigned at least a predetermined minimum number of points as a representative of a respective recognized object (see DOEMLING: e.g., --[0040] In an example, the Gaussian Mixture Model for the point cluster may be generated by using Gaussian Mixture Models of geometric primitives to approximate a shape of the point cluster. It is noted that the shape of the point cluster may be determined by using the geometric information of the point cluster or other information of the point cluster. The Gaussian Mixture Model for the point cluster may be generated by using Gaussian Mixture Models for geometric primitives alone or in combination. As an example, a point cluster has a shape approximating that of a planar ellipse primitive. This shape may be generated by using the Gaussian Mixture Model for the planar ellipse primitive based on geometric parameters of the point cluster. As another example, a point cluster has a shape approximating that of a triangle primitive and a line segment primitive. This shape may be generated by using the combination of a Gaussian Mixture Model for the triangle primitive and a Gaussian Mixture Model for the line segment primitive based on geometric parameters of the point cluster.
[0041] A Gaussian Mixture Model of a geometric primitive may comprise one or more Gaussian distributions, with each Gaussian distribution having a specific mean, a covariance and a composition weight. The number of Gaussian distributions in the Gaussian Mixture Model may be selected based on accuracy requirement. The Gaussian Mixture Models for geometric primitives may be expressed by the geometric parameters of the geometric primitives.
[0042] The Gaussian Mixture Models for point clusters may be generated for example via a substitution method, i.e., by substituting geometric parameters of geometric primitives for concrete geometric parameters of the point clusters. In the case that a point cluster has a shape approximating that of several geometric primitives in combination, generating the Gaussian Mixture Model for the point cluster may also comprise combining Gaussian Mixture Models for the geometric primitives to generate the Gaussian Mixture Model for the point cluster.
[0043] In an example, calculating a Gaussian Mixture Model of a geometric primitive may comprises sampling the geometric primitive and then using data points resulting from the sampling to determine parameters of each Gaussian distribution, such as the specific mean, covariance and composition weight thereof. For example, the specific mean, covariance and composition weight of the Gaussian distribution may be calculated for example by using Expectation Maximization (EM) Algorithm.-- , in [0040]-[0043]; also see EBRAHIMI AFROUZI: e.g., -- the processor of the robot may perform segmentation wherein an object captured in an image is separated from other objects and the background of the image.--, in [0936], and, --the processor may convert the a priori probability P(ω.sub.j) to an a posteriori probability P(x), i.e., the probability of a state of the object being of given the feature value x has been observed. P(ω.sub.j) is the probability of observing the feature value x given the state of the object is ω.sub.j. The product of P(ω.sub.j)P(ω.sub.j) is a significant factor in determining the a posteriori probability whereas the evidence P(x) is a normalizer to ensure the a posteriori probabilities sum to one. In some embodiments, the processor considers more than one feature by replacing the scalar feature value x by a feature vector x, wherein x is of a multi-dimensional Euclidean space R.sup.n or otherwise the feature space. For the feature vector x, a n-component vector-valued random variable, P(ω.sub.j) is the state-conditional probability density function for x, i.e., the probability density function for x conditioned on ω.sub.j being the true category.-- ,in [01071]-[0174]).
Re Claim 2, DOEMLING as modified by EBRAHIMI AFROUZI further disclose wherein, for each of the points of the point cloud, the at least one of the m variables specifies a position, projected in a selected fixed spatial direction, of this point in this spatial direction (see DOEMLING: e.g., --[0024] A point cloud is a set of data points in space. The point cloud is generally produced by a sensor, which measures a large number of points on the external surfaces of objects around it. Each point within the point cloud may have a three dimensional coordinate (x, y, z) and other information (e.g. the color information, the reflection intensity, the normal direction, etc.). The point cloud may be used for many purposes, including creating 3D maps, object recognition and object tracking. In order to achieve these purposes, the point cloud should first be segmented based on some attributes of the points within the point cloud to determine which points belong to a particular object. Due to some reasons such as the limitation of the field of view of the sensor, one frame of point cloud may not obtain the point cloud of the whole object. Therefore, a plurality of consecutive frames of point cloud, which are obtained by the sensor sequentially in time, are typically used for processing.--, in [0024]-[0025]).
Re Claim 8, DOEMLING as modified by EBRAHIMI AFROUZI further disclose wherein m > 1 and at least one of the m variables for each of the points of the point cloud indicates a temperature value or a color value. (see DOEMLING: e.g., Fig. 1, and, --[0024] A point cloud is a set of data points in space. The point cloud is generally produced by a sensor, which measures a large number of points on the external surfaces of objects around it. Each point within the point cloud may have a three dimensional coordinate (x, y, z) and other information (e.g. the color information, the reflection intensity, the normal direction, etc.). The point cloud may be used for many purposes, including creating 3D maps, object recognition and object tracking. In order to achieve these purposes, the point cloud should first be segmented based on some attributes of the points within the point cloud to determine which points belong to a particular object. Due to some reasons such as the limitation of the field of view of the sensor, one frame of point cloud may not obtain the point cloud of the whole object.--, in [0024]-[0025]).
Re Claim 9, DOEMLING as modified by EBRAHIMI AFROUZI further disclose wherein output data are generated and these represent the result of the implemented assignment of the points to segments or the identification of at least one recognized object (see DOEMLING: e.g., -- 3D scanners (e.g. lidar, stereo-camera, time-of-flight camera, etc.) are now widely used in automatic driving systems. Such 3D scanners measure a large number of points on an object's surface, and often output a point cloud as a data file. The point cloud represents the set of points that the 3D scanners has measured. As is known in the art, the point cloud may be used for many purposes, including creating 3D maps, object recognition and object tracking. In order to achieve these purposes, point cloud should first be segmented based on some attributes of the points within the point cloud to obtain clusters of point cloud. The resulting clusters of point cloud can then be described using a suitable model so as to represent a particular object.--, in [0002], and, Fig. 1, and, --[0024] A point cloud is a set of data points in space. The point cloud is generally produced by a sensor, which measures a large number of points on the external surfaces of objects around it. Each point within the point cloud may have a three dimensional coordinate (x, y, z) and other information (e.g. the color information, the reflection intensity, the normal direction, etc.). The point cloud may be used for many purposes, including creating 3D maps, object recognition and object tracking. In order to achieve these purposes, the point cloud should first be segmented based on some attributes of the points within the point cloud to determine which points belong to a particular object. Due to some reasons such as the limitation of the field of view of the sensor, one frame of point cloud may not obtain the point cloud of the whole object.--, in [0024]-[0025]) in one or more of the following ways:
for at least one of the objects, the output data represent an image representation of this object on the basis of one or more of the points of the point cloud which were assigned to the segment belonging to this object (see DOEMLING: e.g., -- 3D scanners (e.g. lidar, stereo-camera, time-of-flight camera, etc.) are now widely used in automatic driving systems. Such 3D scanners measure a large number of points on an object's surface, and often output a point cloud as a data file. The point cloud represents the set of points that the 3D scanners has measured. As is known in the art, the point cloud may be used for many purposes, including creating 3D maps, object recognition and object tracking. In order to achieve these purposes, point cloud should first be segmented based on some attributes of the points within the point cloud to obtain clusters of point cloud. The resulting clusters of point cloud can then be described using a suitable model so as to represent a particular object.--, in [0002], and, Fig. 1, and, --[0024] A point cloud is a set of data points in space. The point cloud is generally produced by a sensor, which measures a large number of points on the external surfaces of objects around it. Each point within the point cloud may have a three dimensional coordinate (x, y, z) and other information (e.g. the color information, the reflection intensity, the normal direction, etc.). The point cloud may be used for many purposes, including creating 3D maps, object recognition and object tracking. In order to achieve these purposes, the point cloud should first be segmented based on some attributes of the points within the point cloud to determine which points belong to a particular object. Due to some reasons such as the limitation of the field of view of the sensor, one frame of point cloud may not obtain the point cloud of the whole object.--, in [0024]-[0025]; and, --the output device 806 may comprise a display for displaying the map, the location mark of the vehicle and also images indicating the travelling situation of the vehicle.--, in [0074]);
the output data represent a piece of information that indicates how many different objects were recognized in the image by means of the segment assignment of the points (see DOEMLING: e.g., -- 3D scanners (e.g. lidar, stereo-camera, time-of-flight camera, etc.) are now widely used in automatic driving systems. Such 3D scanners measure a large number of points on an object's surface, and often output a point cloud as a data file. The point cloud represents the set of points that the 3D scanners has measured. As is known in the art, the point cloud may be used for many purposes, including creating 3D maps, object recognition and object tracking. In order to achieve these purposes, point cloud should first be segmented based on some attributes of the points within the point cloud to obtain clusters of point cloud. The resulting clusters of point cloud can then be described using a suitable model so as to represent a particular object.--, in [0002], and, Fig. 1, and, --[0024] A point cloud is a set of data points in space. The point cloud is generally produced by a sensor, which measures a large number of points on the external surfaces of objects around it. Each point within the point cloud may have a three dimensional coordinate (x, y, z) and other information (e.g. the color information, the reflection intensity, the normal direction, etc.). The point cloud may be used for many purposes, including creating 3D maps, object recognition and object tracking. In order to achieve these purposes, the point cloud should first be segmented based on some attributes of the points within the point cloud to determine which points belong to a particular object. Due to some reasons such as the limitation of the field of view of the sensor, one frame of point cloud may not obtain the point cloud of the whole object.--, in [0024]-[0025]; and, --the output device 806 may comprise a display for displaying the map, the location mark of the vehicle and also images indicating the travelling situation of the vehicle.--, in [0074]);;
the output data represent a piece of information that indicates the respective segment or object to which the points were assigned in each case (see DOEMLING: e.g., -- 3D scanners (e.g. lidar, stereo-camera, time-of-flight camera, etc.) are now widely used in automatic driving systems. Such 3D scanners measure a large number of points on an object's surface, and often output a point cloud as a data file. The point cloud represents the set of points that the 3D scanners has measured. As is known in the art, the point cloud may be used for many purposes, including creating 3D maps, object recognition and object tracking. In order to achieve these purposes, point cloud should first be segmented based on some attributes of the points within the point cloud to obtain clusters of point cloud. The resulting clusters of point cloud can then be described using a suitable model so as to represent a particular object.--, in [0002], and, Fig. 1, and, --[0024] A point cloud is a set of data points in space. The point cloud is generally produced by a sensor, which measures a large number of points on the external surfaces of objects around it. Each point within the point cloud may have a three dimensional coordinate (x, y, z) and other information (e.g. the color information, the reflection intensity, the normal direction, etc.). The point cloud may be used for many purposes, including creating 3D maps, object recognition and object tracking. In order to achieve these purposes, the point cloud should first be segmented based on some attributes of the points within the point cloud to determine which points belong to a particular object. Due to some reasons such as the limitation of the field of view of the sensor, one frame of point cloud may not obtain the point cloud of the whole object.--, in [0024]-[0025]; and, --the output device 806 may comprise a display for displaying the map, the location mark of the vehicle and also images indicating the travelling situation of the vehicle.--, in [0074]);;
the output data represent a piece of information that, for at least a subset of the points, specifies the respective function value of one or more of the probability density functions at the location determined by the values of the m variables assigned to the point (see DOEMLING: e.g., -- 3D scanners (e.g. lidar, stereo-camera, time-of-flight camera, etc.) are now widely used in automatic driving systems. Such 3D scanners measure a large number of points on an object's surface, and often output a point cloud as a data file. The point cloud represents the set of points that the 3D scanners has measured. As is known in the art, the point cloud may be used for many purposes, including creating 3D maps, object recognition and object tracking. In order to achieve these purposes, point cloud should first be segmented based on some attributes of the points within the point cloud to obtain clusters of point cloud. The resulting clusters of point cloud can then be described using a suitable model so as to represent a particular object.--, in [0002], and, Fig. 1, and, --[0024] A point cloud is a set of data points in space. The point cloud is generally produced by a sensor, which measures a large number of points on the external surfaces of objects around it. Each point within the point cloud may have a three dimensional coordinate (x, y, z) and other information (e.g. the color information, the reflection intensity, the normal direction, etc.). The point cloud may be used for many purposes, including creating 3D maps, object recognition and object tracking. In order to achieve these purposes, the point cloud should first be segmented based on some attributes of the points within the point cloud to determine which points belong to a particular object. Due to some reasons such as the limitation of the field of view of the sensor, one frame of point cloud may not obtain the point cloud of the whole object.--, in [0024]-[0025]; and, --the output device 806 may comprise a display for displaying the map, the location mark of the vehicle and also images indicating the travelling situation of the vehicle.--, in [0074]).
Re Claim 10, DOEMLING as modified by EBRAHIMI AFROUZI further disclose wherein, for at least one of the m variables, the associated probability density functions each have a curve where the function value, as a function of the value of the variable, increases up to a maximum and then falls again, the maximum being the only maximum that occurs in the curve of the probability density function (see DOEMLING: e.g., --[0026] As shown on the left side of FIG. 1, the first frame of point cloud “Frame 1” may be obtained by the sensor at time t.sub.1. In Frame 1, only 24 points are depicted for ease of illustration. It should be noted that a frame of point cloud may contain hundreds or thousands of points. In order to segment Frame 1, various well-known segmentation algorithms may be employed. These segmentation algorithms may include the K-means algorithm, the K-NearestNeighbor algorithm, the GMM algorithm and/or the region growing algorithm. After the segmentation, Frame 1 may be segmented into three point clusters which are shown within the dashed circles 110, 112 and 114 in FIG. 1. The first point cluster 110 may include 18 points. The second point cluster 112 may include 4 points. The third pint cluster 114 may include 2 points. Each point cluster may have attributes such as length, width, thickness, orientation, or the ratio of main axes, and may indicate that points within the point cluster belong to the same object.
[0027] At time t.sub.2, the sensor may, for example, have been rotated for certain degree in the X-Y plane in the Cartesian coordinate system since t.sub.1 and may obtain the second frame of point cloud “Frame 2”. Frame 2 may also be segmented using abovementioned segmentation algorithms and may be segmented into three point clusters 120, 122 and 124. The fourth point cluster 120 may include 9 points. The fifth point cluster 122 may include 9 points. The sixth point cluster 124 may include 2 points.--, in [0026]-[0027], and, -- a method for representing a point cluster may comprise (a) generating a Gaussian Mixture Model for the point cluster; (b) generating a signature for identifying the point cluster; and (c) generating a SGM model for representing the point cluster.
[0036] A Gaussian Mixture Model is a type of density model which comprises a number of Gaussian distributions. For example, a typical Gaussian Mixture Model in one dimension may take the form of formula (1) as below:
p(x)=Σ.sub.k=1.sup.Kπ.sub.kp(x|k) (1)
wherein K is the number of Gaussian distributions contained in the Gaussian Mixture Model, π.sub.k is the composition weight of the k.sup.th Gaussian distribution, and p(x|k) is the probability density function of the k.sup.th Gaussian distribution which is defined by its average μ.sub.k and covariance b.sub.k. In particular, a typical Gaussian distribution (i.e. the probability density function of the Gaussian distribution) in one dimension may take the form of formula (2)--, in [0035]-[0039], and, --[0040] In an example, the Gaussian Mixture Model for the point cluster may be generated by using Gaussian Mixture Models of geometric primitives to approximate a shape of the point cluster. It is noted that the shape of the point cluster may be determined by using the geometric information of the point cluster or other information of the point cluster. The Gaussian Mixture Model for the point cluster may be generated by using Gaussian Mixture Models for geometric primitives alone or in combination. As an example, a point cluster has a shape approximating that of a planar ellipse primitive. This shape may be generated by using the Gaussian Mixture Model for the planar ellipse primitive based on geometric parameters of the point cluster. As another example, a point cluster has a shape approximating that of a triangle primitive and a line segment primitive. This shape may be generated by using the combination of a Gaussian Mixture Model for the triangle primitive and a Gaussian Mixture Model for the line segment primitive based on geometric parameters of the point cluster.
[0041] A Gaussian Mixture Model of a geometric primitive may comprise one or more Gaussian distributions, with each Gaussian distribution having a specific mean, a covariance and a composition weight. The number of Gaussian distributions in the Gaussian Mixture Model may be selected based on accuracy requirement. The Gaussian Mixture Models for geometric primitives may be expressed by the geometric parameters of the geometric primitives.
[0042] The Gaussian Mixture Models for point clusters may be generated for example via a substitution method, i.e., by substituting geometric parameters of geometric primitives for concrete geometric parameters of the point clusters. In the case that a point cluster has a shape approximating that of several geometric primitives in combination, generating the Gaussian Mixture Model for the point cluster may also comprise combining Gaussian Mixture Models for the geometric primitives to generate the Gaussian Mixture Model for the point cluster.
[0043] In an example, calculating a Gaussian Mixture Model of a geometric primitive may comprises sampling the geometric primitive and then using data points resulting from the sampling to determine parameters of each Gaussian distribution, such as the specific mean, covariance and composition weight thereof. For example, the specific mean, covariance and composition weight of the Gaussian distribution may be calculated for example by using Expectation Maximization (EM) Algorithm.-- , in [0040]-[0043]).
Re Claim 11, DOEMLING as modified by EBRAHIMI AFROUZI further disclose wherein at least one of the respective probability density functions for at least one of the m variables is a Gaussian function (see DOEMLING: e.g., --[0026] As shown on the left side of FIG. 1, the first frame of point cloud “Frame 1” may be obtained by the sensor at time t.sub.1. In Frame 1, only 24 points are depicted for ease of illustration. It should be noted that a frame of point cloud may contain hundreds or thousands of points. In order to segment Frame 1, various well-known segmentation algorithms may be employed. These segmentation algorithms may include the K-means algorithm, the K-NearestNeighbor algorithm, the GMM algorithm and/or the region growing algorithm. After the segmentation, Frame 1 may be segmented into three point clusters which are shown within the dashed circles 110, 112 and 114 in FIG. 1. The first point cluster 110 may include 18 points. The second point cluster 112 may include 4 points. The third pint cluster 114 may include 2 points. Each point cluster may have attributes such as length, width, thickness, orientation, or the ratio of main axes, and may indicate that points within the point cluster belong to the same object.
[0027] At time t.sub.2, the sensor may, for example, have been rotated for certain degree in the X-Y plane in the Cartesian coordinate system since t.sub.1 and may obtain the second frame of point cloud “Frame 2”. Frame 2 may also be segmented using abovementioned segmentation algorithms and may be segmented into three point clusters 120, 122 and 124. The fourth point cluster 120 may include 9 points. The fifth point cluster 122 may include 9 points. The sixth point cluster 124 may include 2 points.--, in [0026]-[0027], and, -- a method for representing a point cluster may comprise (a) generating a Gaussian Mixture Model for the point cluster; (b) generating a signature for identifying the point cluster; and (c) generating a SGM model for representing the point cluster.
[0036] A Gaussian Mixture Model is a type of density model which comprises a number of Gaussian distributions. For example, a typical Gaussian Mixture Model in one dimension may take the form of formula (1) as below:
p(x)=Σ.sub.k=1.sup.Kπ.sub.kp(x|k) (1)
wherein K is the number of Gaussian distributions contained in the Gaussian Mixture Model, π.sub.k is the composition weight of the k.sup.th Gaussian distribution, and p(x|k) is the probability density function of the k.sup.th Gaussian distribution which is defined by its average μ.sub.k and covariance b.sub.k. In particular, a typical Gaussian distribution (i.e. the probability density function of the Gaussian distribution) in one dimension may take the form of formula (2)--, in [0035]-[0039], and, --[0040] In an example, the Gaussian Mixture Model for the point cluster may be generated by using Gaussian Mixture Models of geometric primitives to approximate a shape of the point cluster. It is noted that the shape of the point cluster may be determined by using the geometric information of the point cluster or other information of the point cluster. The Gaussian Mixture Model for the point cluster may be generated by using Gaussian Mixture Models for geometric primitives alone or in combination. As an example, a point cluster has a shape approximating that of a planar ellipse primitive. This shape may be generated by using the Gaussian Mixture Model for the planar ellipse primitive based on geometric parameters of the point cluster. As another example, a point cluster has a shape approximating that of a triangle primitive and a line segment primitive. This shape may be generated by using the combination of a Gaussian Mixture Model for the triangle primitive and a Gaussian Mixture Model for the line segment primitive based on geometric parameters of the point cluster.
[0041] A Gaussian Mixture Model of a geometric primitive may comprise one or more Gaussian distributions, with each Gaussian distribution having a specific mean, a covariance and a composition weight. The number of Gaussian distributions in the Gaussian Mixture Model may be selected based on accuracy requirement. The Gaussian Mixture Models for geometric primitives may be expressed by the geometric parameters of the geometric primitives.
[0042] The Gaussian Mixture Models for point clusters may be generated for example via a substitution method, i.e., by substituting geometric parameters of geometric primitives for concrete geometric parameters of the point clusters. In the case that a point cluster has a shape approximating that of several geometric primitives in combination, generating the Gaussian Mixture Model for the point cluster may also comprise combining Gaussian Mixture Models for the geometric primitives to generate the Gaussian Mixture Model for the point cluster.
[0043] In an example, calculating a Gaussian Mixture Model of a geometric primitive may comprises sampling the geometric primitive and then using data points resulting from the sampling to determine parameters of each Gaussian distribution, such as the specific mean, covariance and composition weight thereof. For example, the specific mean, covariance and composition weight of the Gaussian distribution may be calculated for example by using Expectation Maximization (EM) Algorithm.-- , in [0040]-[0043]).
Re Claim 12, DOEMLING as modified by EBRAHIMI AFROUZI further disclose wherein at least one of the frequency distributions is subjected to a respective smoothing process and the approximation with regard to this at least one frequency distribution is implemented with regard to the corresponding frequency distribution smoothed by means of the smoothing process (see EBRAHIMI AFROUZI: e.g., --the processor may use shape descriptors for objects. In embodiments, shape descriptors are immune to rotation, translation, and scaling. In embodiments, shape descriptors may be region based descriptors or boundary based descriptors. In some embodiments, the processor may use curvature Fourier descriptors wherein the image contour is extracted by sampling coordinates along the contour, the coordinates of the sample being S={s.sub.1(x.sub.1,y.sub.1), s.sub.2(x.sub.2,y.sub.2), . . . , s.sub.n(x.sub.n,y.sub.n)}. The contour may then be smoothened using, for example, a Gaussian with different standard deviation. The image may then be scaled and the Fourier transform applied. In some embodiments, the processor describes any continuous curve ƒ(t)=(x y).sub.t=(ƒ.sub.x(t) ƒ.sub.y(t)), wherein 0<t<t.sub.max and t is the path length along the curvature. Sampling a curve uniformly creates a set that is infinite and periodic. To create a sequence, the processor selects an arbitrary point g.sub.1 in the contour with a position (x.sub.0 y.sub.0) and continues to sample points with different x,y positions along the path of the contour at equal distance steps.--, in [0947]).
Re Claim 13, DOEMLING as modified by EBRAHIMI AFROUZI further disclose wherein a gesture recognition process is carried out on the basis of the respective points of one or more of the segments identified as a representative of a respective object, in order to recognize a person's gesture imaged in the image by means of the point cloud (see EBRAHIMI AFROUZI: e.g., -- [0957] In embodiments, object tracking may be challenging when the robot is on the move. With the robot, its sensing devices are moving, and in some cases, the object being tracked is moving as well. In some embodiments, the processor may track movement of a non-SLAM enabled object within a scene by detecting a presence of the object in a previous act of sensing and its lack of presence in a current act of sensing and vice versa. A displacement of the object in an act of sensing (e.g., a captured image) that does not correspond to what is expected or predicted based on movement of the robot may also be used by the processor as an indication of a moving object. In some embodiments, the processor may be interested in more than just the presence of the object. For example, the processor of the robot may be interested in understanding a hand gesture, such as an instruction to stop or navigate to a certain place given by a hand gesture such as finger pointing.--, in [0957]).
Re Claim 14, claim 14 is the corresponding system claim to claim 1 respectively. Thus, claim 14 rejected for the similar reasons as for claim 1. Furthermore, DOEMLING as modified by EBRAHIMI AFROUZI further disclose a data processing system having at least one processor configured to carry out the method (see DOEMLING: e.g., -- 3D scanners (e.g. lidar, stereo-camera, time-of-flight camera, etc.) are now widely used in automatic driving systems. Such 3D scanners measure a large number of points on an object's surface, and often output a point cloud as a data file. The point cloud represents the set of points that the 3D scanners has measured. As is known in the art, the point cloud may be used for many purposes, including creating 3D maps, object recognition and object tracking. In order to achieve these purposes, point cloud should first be segmented based on some attributes of the points within the point cloud to obtain clusters of point cloud. The resulting clusters of point cloud can then be described using a suitable model so as to represent a particular object.--, in [0002], -- there is provided an apparatus for segmentation of a point cloud, comprising: a memory with computer executable instructions stored therein; and a processor, coupled to the memory and configured to: receive a first frame of point cloud from a sensor; segment the first frame of point cloud using a segmentation algorithm to obtain a first set of point clusters representing a segmentation result for the first frame of point cloud; receive a second frame of point cloud from the sensor; map the first set of point clusters to the second frame of point cloud; determine points within the second frame of point cloud which do not belong to the mapped first set of point clusters; segment the points within the second frame of point cloud which do not belong to the mapped first set of point clusters using the segmentation algorithm to obtain a second set of point clusters; and generate a segmentation result for the second frame of point cloud by combining the first set of point clusters and the second set of point clusters.--, in [0008]; and, Fig. 1, and, --[0024] A point cloud is a set of data points in space. The point cloud is generally produced by a sensor, which measures a large number of points on the external surfaces of objects around it. Each point within the point cloud may have a three dimensional coordinate (x, y, z) and other information (e.g. the color information, the reflection intensity, the normal direction, etc.). The point cloud may be used for many purposes, including creating 3D maps, object recognition and object tracking. In order to achieve these purposes, the point cloud should first be segmented based on some attributes of the points within the point cloud to determine which points belong to a particular object. Due to some reasons such as the limitation of the field of view of the sensor, one frame of point cloud may not obtain the point cloud of the whole object.--, in [0024]-[0025]).
Claims 3-7 are rejected under 35 U.S.C. 103 as being patentable over DOEMLING as modified by EBRAHIMI AFROUZI, and further in view of ECKART (US 20200372649 A1, Date Filed: 2018-11-05).
Re Claim 3, DOEMLING as modified by EBRAHIMI AFROUZI further disclose wherein the fixed spatial direction is selected so as to run orthogonally to a plane (see EBRAHIMI AFROUZI: e.g., --The virtual camera may include properties of the real camera, such as position and orientation defined by a point coordinate and a direction vector and lens and focal point which together define the perspective distortion of the resulting images. With zero distortion, an orthographic view of the model is obtained, wherein objects remain a same size regardless of their distance from the camera. Orthographic views may appear unrealistic, especially for larger models, however, they are useful for measuring and giving an overall understanding of the model. Examples of orthographic views include isometric, dimetric, and trimetric, as illustrated in FIG. 97. As the orientation of the camera (and therefore the viewing plane) changes, these orthographic views may be converted from one to another. In some embodiments, an oblique projection may be used. In embodiments, an oblique projection may appear even less realistic compared to orthographic projection.--, in [0920], and, -- Effective focal length is the distance between focal point and its corresponding principal point (center of principal plane). The principal planes are two hypothetical planes in a lens system at which all the refraction can be considered to happen. For a given set of lenses and separations, the principal planes are fixed and do not depend upon the object position.--, in [1200]);
DOEMLING as modified by EBRAHIMI AFROUZI however do not explicitly disclose the fixed spatial direction is selected so as to run orthogonally to a plane a first principal component that emerges from a principal component analysis applied to the point cloud,
ECKART discloses the fixed spatial direction is selected so as to run orthogonally to a plane a first principal component that emerges from a principal component analysis applied to the point cloud (see ECKART: e.g., -- reinterpreting the Mahalanobis distance calculation can lead to a highly accurate technique for determining the estimated transformation T that minimizes the MLE criteria. More specifically, the inner Mahalanobis distance term can be rewritten by decomposing each covariance matrix into its associated eigenvalues and eigenvectors using Principal Component Analysis (PCA), and interpreting each Mahalanobis distance term as a weighted sum of three separate point-to-plane distances, where the weights are inversely proportional to the eigenvalues associated with each plane defined by a corresponding eigenvector…
[0158] FIG. 9 illustrates the M step calculations for a single mixture component, in accordance with some embodiments. For each mixture component in the hierarchical tree data structure 700, calculate a first parameter π.sub.j* for the mixture component 910 in accordance with Equation 10 and a second parameter μ.sub.j* for the mixture component 910 in accordance with Equation 11. The first parameter is the average weight associated with the mixture component for the N points in the point cloud being registered. The second parameter is a mean location for the N points in the point cloud, with respect to the weights such that the mean location is skewed toward the mixture component based on the probability associated with each point. As shown in FIG. 9, three point objects 902, 904, and 906 can contribute to the calculation of the first and second parameters. The first and second parameters are based on the correspondence values (e.g., γ.sub.ij) determined during the E step.
[0159] The inner sum from Equation 13 can then be calculated for the mixture component based on the eigenvectors 922, 924 (illustrated in only two dimensions) and the mean μ.sub.j of the mixture component 912. In other words, the M step first locates a position relative to the mean of the mixture component 912, based on the correspondence values for the N points in the point cloud and the location of those points, and then the M step calculates a squared Mahalanobis distance for that position along the three principal axes of the covariance based on varying the transformation T. --, in [0156]-[0159]);
DOEMLING (as modified by EBRAHIMI AFROUZI) and ECKART are combinable as they are in the same field of endeavor: recognizing objects in the image. Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to further modify DOEMLING (as modified by EBRAHIMI AFROUZI)’s method using ECKART’s teachings by including the fixed spatial direction is selected so as to run orthogonally to a plane a first principal component that emerges from a principal component analysis applied to the point cloud to DOEMLING (as modified by EBRAHIMI AFROUZI)’s point cloud processing in order to perform object/scene recognition with aligning two or more point clouds by estimating relative transformations between the point clouds (see ECKART: e.g. in abstract, and [0156]-[0159]).
Re Claim 4, DOEMLING as modified by EBRAHIMI AFROUZI and ECKART further disclose wherein M € {2; 3} and the fixed spatial direction is selected so that it corresponds to the second principal component arising from the principal component analysis in the case M=2 and to the third principal component arising from the principal component analysis in the case M=3 (see ECKART: e.g., -- reinterpreting the Mahalanobis distance calculation can lead to a highly accurate technique for determining the estimated transformation T that minimizes the MLE criteria. More specifically, the inner Mahalanobis distance term can be rewritten by decomposing each covariance matrix into its associated eigenvalues and eigenvectors using Principal Component Analysis (PCA), and interpreting each Mahalanobis distance term as a weighted sum of three separate point-to-plane distances, where the weights are inversely proportional to the eigenvalues associated with each plane defined by a corresponding eigenvector…
[0158] FIG. 9 illustrates the M step calculations for a single mixture component, in accordance with some embodiments. For each mixture component in the hierarchical tree data structure 700, calculate a first parameter π.sub.j* for the mixture component 910 in accordance with Equation 10 and a second parameter μ.sub.j* for the mixture component 910 in accordance with Equation 11. The first parameter is the average weight associated with the mixture component for the N points in the point cloud being registered. The second parameter is a mean location for the N points in the point cloud, with respect to the weights such that the mean location is skewed toward the mixture component based on the probability associated with each point. As shown in FIG. 9, three point objects 902, 904, and 906 can contribute to the calculation of the first and second parameters. The first and second parameters are based on the correspondence values (e.g., γ.sub.ij) determined during the E step.
[0159] The inner sum from Equation 13 can then be calculated for the mixture component based on the eigenvectors 922, 924 (illustrated in only two dimensions) and the mean μ.sub.j of the mixture component 912. In other words, the M step first locates a position relative to the mean of the mixture component 912, based on the correspondence values for the N points in the point cloud and the location of those points, and then the M step calculates a squared Mahalanobis distance for that position along the three principal axes of the covariance based on varying the transformation T. --, in [0156]-[0159]).
Re Claim 5, DOEMLING as modified by EBRAHIMI AFROUZI and ECKART further disclose filtering the image so that post filtering it only still contains those points of the point of the point cloud which were assigned to one of the segments respectively identified as a representative of a respective recognized object (see ECKART: e.g., -- the point cloud 600 is generated by one or more active sensors such as depth cameras, LiDAR, or other similar technology. The active sensors can be configured to scan a space and create the point objects 601 in the point cloud 600. The number of point objects 601 included in the point cloud 600 can be very large, numbering in the hundreds of thousands, millions, or even billions of point objects 601. In some embodiments, the size of a point cloud 600 can be limited, either by the configuration of the active sensors or by filtering the point objects 601 added to the point cloud 600. Limiting the number of point objects 601 included in a point cloud 600 can be necessary to reduce the time to process the point cloud 600 for time-sensitive tasks.--, in [0111]). See the similar obviousness and motivation statements for the combination of the cited references as addressed above for claim 4.
Re Claim 6, DOEMLING as modified by EBRAHIMI AFROUZI and ECKART further disclose wherein the image is filtered so that post filtering it only still contains those points of the point cloud which were assigned to exactly one specific selected segment of those segments identified as a representative of an assigned recognized object (see ECKART: e.g., -- the point cloud 600 is generated by one or more active sensors such as depth cameras, LiDAR, or other similar technology. The active sensors can be configured to scan a space and create the point objects 601 in the point cloud 600. The number of point objects 601 included in the point cloud 600 can be very large, numbering in the hundreds of thousands, millions, or even billions of point objects 601. In some embodiments, the size of a point cloud 600 can be limited, either by the configuration of the active sensors or by filtering the point objects 601 added to the point cloud 600. Limiting the number of point objects 601 included in a point cloud 600 can be necessary to reduce the time to process the point cloud 600 for time-sensitive tasks.--, in [0111]).
Re Claim 7, DOEMLING as modified by EBRAHIMI AFROUZI and ECKART further disclose wherein m = 1 and the segment selected from the set of segments identified in each case as a representative of a respective recognized object is the segment whose assigned points, in accordance with their positions projected on the selected fixed spatial direction, when viewed in this spatial direction as direction of view, and when considered on average, are closer than the points assigned to any other of the identified segments (see DOEMLING: e.g., -- 3D scanners (e.g. lidar, stereo-camera, time-of-flight camera, etc.) are now widely used in automatic driving systems. Such 3D scanners measure a large number of points on an object's surface, and often output a point cloud as a data file. The point cloud represents the set of points that the 3D scanners has measured. As is known in the art, the point cloud may be used for many purposes, including creating 3D maps, object recognition and object tracking. In order to achieve these purposes, point cloud should first be segmented based on some attributes of the points within the point cloud to obtain clusters of point cloud. The resulting clusters of point cloud can then be described using a suitable model so as to represent a particular object.--, in [0002], and, Fig. 1, and, --[0024] A point cloud is a set of data points in space. The point cloud is generally produced by a sensor, which measures a large number of points on the external surfaces of objects around it. Each point within the point cloud may have a three dimensional coordinate (x, y, z) and other information (e.g. the color information, the reflection intensity, the normal direction, etc.). The point cloud may be used for many purposes, including creating 3D maps, object recognition and object tracking. In order to achieve these purposes, the point cloud should first be segmented based on some attributes of the points within the point cloud to determine which points belong to a particular object. Due to some reasons such as the limitation of the field of view of the sensor, one frame of point cloud may not obtain the point cloud of the whole object.--, in [0024]-[0025]; and, --[0040] In an example, the Gaussian Mixture Model for the point cluster may be generated by using Gaussian Mixture Models of geometric primitives to approximate a shape of the point cluster. It is noted that the shape of the point cluster may be determined by using the geometric information of the point cluster or other information of the point cluster. The Gaussian Mixture Model for the point cluster may be generated by using Gaussian Mixture Models for geometric primitives alone or in combination. As an example, a point cluster has a shape approximating that of a planar ellipse primitive. This shape may be generated by using the Gaussian Mixture Model for the planar ellipse primitive based on geometric parameters of the point cluster. As another example, a point cluster has a shape approximating that of a triangle primitive and a line segment primitive. This shape may be generated by using the combination of a Gaussian Mixture Model for the triangle primitive and a Gaussian Mixture Model for the line segment primitive based on geometric parameters of the point cluster.
[0041] A Gaussian Mixture Model of a geometric primitive may comprise one or more Gaussian distributions, with each Gaussian distribution having a specific mean, a covariance and a composition weight. The number of Gaussian distributions in the Gaussian Mixture Model may be selected based on accuracy requirement. The Gaussian Mixture Models for geometric primitives may be expressed by the geometric parameters of the geometric primitives.
[0042] The Gaussian Mixture Models for point clusters may be generated for example via a substitution method, i.e., by substituting geometric parameters of geometric primitives for concrete geometric parameters of the point clusters. In the case that a point cluster has a shape approximating that of several geometric primitives in combination, generating the Gaussian Mixture Model for the point cluster may also comprise combining Gaussian Mixture Models for the geometric primitives to generate the Gaussian Mixture Model for the point cluster.
[0043] In an example, calculating a Gaussian Mixture Model of a geometric primitive may comprises sampling the geometric primitive and then using data points resulting from the sampling to determine parameters of each Gaussian distribution, such as the specific mean, covariance and composition weight thereof. For example, the specific mean, covariance and composition weight of the Gaussian distribution may be calculated for example by using Expectation Maximization (EM) Algorithm.-- , in [0040]-[0043]; and,
see EBRAHIMI AFROUZI: e.g., --The virtual camera may include properties of the real camera, such as position and orientation defined by a point coordinate and a direction vector and lens and focal point which together define the perspective distortion of the resulting images. With zero distortion, an orthographic view of the model is obtained, wherein objects remain a same size regardless of their distance from the camera. Orthographic views may appear unrealistic, especially for larger models, however, they are useful for measuring and giving an overall understanding of the model. Examples of orthographic views include isometric, dimetric, and trimetric, as illustrated in FIG. 97. As the orientation of the camera (and therefore the viewing plane) changes, these orthographic views may be converted from one to another. In some embodiments, an oblique projection may be used. In embodiments, an oblique projection may appear even less realistic compared to orthographic projection.--, in [0920], and, -- Effective focal length is the distance between focal point and its corresponding principal point (center of principal plane). The principal planes are two hypothetical planes in a lens system at which all the refraction can be considered to happen. For a given set of lenses and separations, the principal planes are fixed and do not depend upon the object position.--, in [1200];also see ECKART: e.g., -- reinterpreting the Mahalanobis distance calculation can lead to a highly accurate technique for determining the estimated transformation T that minimizes the MLE criteria. More specifically, the inner Mahalanobis distance term can be rewritten by decomposing each covariance matrix into its associated eigenvalues and eigenvectors using Principal Component Analysis (PCA), and interpreting each Mahalanobis distance term as a weighted sum of three separate point-to-plane distances, where the weights are inversely proportional to the eigenvalues associated with each plane defined by a corresponding eigenvector…
[0158] FIG. 9 illustrates the M step calculations for a single mixture component, in accordance with some embodiments. For each mixture component in the hierarchical tree data structure 700, calculate a first parameter π.sub.j* for the mixture component 910 in accordance with Equation 10 and a second parameter μ.sub.j* for the mixture component 910 in accordance with Equation 11. The first parameter is the average weight associated with the mixture component for the N points in the point cloud being registered. The second parameter is a mean location for the N points in the point cloud, with respect to the weights such that the mean location is skewed toward the mixture component based on the probability associated with each point. As shown in FIG. 9, three point objects 902, 904, and 906 can contribute to the calculation of the first and second parameters. The first and second parameters are based on the correspondence values (e.g., γ.sub.ij) determined during the E step.
[0159] The inner sum from Equation 13 can then be calculated for the mixture component based on the eigenvectors 922, 924 (illustrated in only two dimensions) and the mean μ.sub.j of the mixture component 912. In other words, the M step first locates a position relative to the mean of the mixture component 912, based on the correspondence values for the N points in the point cloud and the location of those points, and then the M step calculates a squared Mahalanobis distance for that position along the three principal axes of the covariance based on varying the transformation T. --, in [0156]-[0159]).
Conclusion
Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WEI WEN YANG whose telephone number is (571)270-5670. The examiner can normally be reached on 8:00 - 5:00 pm.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amandeep Saini can be reached on 571-272-3382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WEI WEN YANG/Primary Examiner, Art Unit 2662