DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). The certified copy has been filed in parent Application No. PCT/EP2022/066050, filed on 06/13/2022.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1, 3, 14, 16 and 18 are rejected under 35 U.S.C. 102 (a)(1) as being anticipated by Zhejiang University (CN 111640152 A).
Regarding Claim 16, Zhejiang University teaches An apparatus (Page 2: “Referring to FIG. 1, the fish growth monitoring system 1 in this embodiment is disposed under an unmanned ship or in a fixed farm”)
for determining a dimension value indicating a physical dimension of a three-dimensional, (3D) space (Page 2: “the real length information of the target fish is acquired by the monitoring system of the present embodiment”),
the apparatus comprising:
a processing circuitry (Page 2: “the fish growth monitoring system 1 in this embodiment is disposed under an unmanned ship or in a fixed farm, and includes … a server (or an embedded processor) communicatively connected to the modules”); and
a memory containing instructions (Page 2: “the fish growth monitoring system 1 in this embodiment is disposed under an unmanned ship or in a fixed farm, and includes … a calculation module 400, and further includes a binocular camera and a server (or an embedded processor) communicatively connected to the modules”. Notes: The system performs a method for calculating a dimension value indicating a physical dimension of a 3D space via a processor and a calculation module; a storage/memory is necessitated for this function given that the system has both a processor and a calculation module) for configuring the apparatus to:
obtain a first image, wherein the first image is generated using a first lens of a camera (Page 2: “After obtaining all pixel coordinates of the fish in the two-dimensional image, stereo matching is performed by using the binocular camera and the SGBM algorithm. Firstly, the inner and outer parameters of the binocular camera under water are obtained by using the calibration plate and the normal friend calibration method, and then the Bouguet algorithm is used for camera picture correction (avoiding distortion)”. Notes: stereo matching requires two images from different viewpoints, which are provided by the binocular camera, which has two image sensors for capturing images);
identify a first set of one or more key points included in the first image (Figure 1 illustrates key point P’ as perceived via the left camera, and P’’ as perceived by the right camera; Page 2: “Solving an edge extraction and communication area of the target box to obtain a set U of all pixels in the target box, where all pixels {x i, y i} in the set U are pixels constituting the fish… performing parallax matching on each pixel in the set U to obtain a disparity value thereof, and performing optimization processing … After obtaining all pixel coordinates of the fish in the two-dimensional image, stereo matching is performed by using the binocular camera and the SGBM algorithm”);
obtain a second image, wherein the second image is generated using a second lens of the camera (Page 2: “After obtaining all pixel coordinates of the fish in the two-dimensional image, stereo matching is performed by using the binocular camera and the SGBM algorithm. Firstly, the inner and outer parameters of the binocular camera under water are obtained by using the calibration plate and the normal friend calibration method, and then the Bouguet algorithm is used for camera picture correction (avoiding distortion)”. Notes: stereo matching requires two images from different viewpoints, which are provided by the binocular camera, which has two image sensors for capturing images);
identify a second set of one or more key points included in the second image (Figure 1 illustrates key point P’ as perceived via the left camera, and P’’ as perceived by the right camera; Page 2: “Solving an edge extraction and communication area of the target box to obtain a set U of all pixels in the target box, where all pixels {x i, y i} in the set U are pixels constituting the fish… performing parallax matching on each pixel in the set U to obtain a disparity value thereof, and performing optimization processing … After obtaining all pixel coordinates of the fish in the two-dimensional image, stereo matching is performed by using the binocular camera and the SGBM algorithm”);
determine a set of one or more 3D points associated with the first set of key points and the second set of key points (Page 2: “obtaining all pixel coordinates of the fish in the two-dimensional image, stereo matching is performed by using the binocular camera and the SGBM algorithm. Firstly, the inner and outer parameters of the binocular camera under water are obtained by using the calibration plate and the normal friend calibration method” … Page 2: “After the binocular camera parameter is acquired, the SBM is used to perform disparity matching calculation on all pixels of the above-mentioned fish pixel point set U to obtain a disparity value d thereof, and after obtaining the disparity value, all the pixels in the U are (x, y, i, d)”. Notes: disparity values are an indicator of depth, and hence after disparity matching, the resulting pixels in U are 3D points),
wherein the set of one or more 3D points includes a first 3D point (Page 2: “After the binocular camera parameter is acquired, the SBM is used to perform disparity matching calculation on all pixels of the above-mentioned fish pixel point set U to obtain a disparity value d thereof, and after obtaining the disparity value, all the pixels in the U are (x, y, i, d));
calculate a first distance value indicating a distance between the camera and a real- world point corresponding to the first 3D point (Page 2: “After the binocular camera parameter is acquired, the SBM is used to perform disparity matching calculation on all pixels of the above-mentioned fish pixel point set U to obtain a disparity value d thereof, and after obtaining the disparity value, all the pixels in the U are (x, y, i, d); Figure 2; Page 2: “Finally, by using the camera parameters and its imaging principle, the distance (depth) depth from the [fish] to the camera is calculated according to the disparity value d of each pixel point, the principle is as shown in FIG. 2, x_left and x_right are coordinates of the real point at the left eye and the right eye camera, B (Baseline) is the camera baseline parameter, O_left and O_right are dual gaze centers, and f is the focal length. The depth may be solved by a formula depth = f * B/(x_left-x_right)”; Page 2: “After the depth of each pixel is obtained, the average distance D from all the pixels in the set U to the camera is obtained by averaging, and the average distance D is used as the distance from the fish to the camera”. Notes: each depth value corresponds with a distance from the camera to a real-world point corresponding to a pixel point in U); and
based at least on the calculated first distance value, determine the dimension value (Page 1: “solving the real length of the fish according to the length of the fish in the target fish image and the distance from the fish calculated in step 4) to the camera, so as to realize fish growth monitoring”).
Claim 1, being similar in scope to Claim 16, is rejected under the same rationale.
Regarding Claim 14, the method of Claim 1 is rejected over Zhejiang University.
Zhejiang University teaches a non-transitory computer readable storage medium storing a computer program comprising instructions for configuring an apparatus to perform the method of Claim 1 (Page 2: “the fish growth monitoring system 1 in this embodiment is disposed under an unmanned ship or in a fixed farm, and includes a target detection module 100, a stereo matching module 200, a backend optimization module 300, and a calculation module 400, and further includes a binocular camera and a server (or an embedded processor) communicatively connected to the modules”. Notes: The system performs a method for calculating a dimension value indicating a physical dimension of a 3D space via a processor and a calculation module; a storage/memory is necessitated for carrying out the instructions for the function given that the system has both a processor and a calculation module).
Regarding Claim 18, the apparatus of Claim 16 is rejected over Zhejiang University.
Zhejiang University teaches the apparatus of Claim 16, wherein the apparatus is further configured to:
identify a first subset of one or more key points from the first set of key points (Figure 1 illustrates key point P’ as perceived via the left camera, and P’’ as perceived by the right camera; Page 2: “Solving an edge extraction and communication area of the target box to obtain a set U of all pixels in the target box, where all pixels {x i, y i} in the set U are pixels constituting the fish… performing parallax matching on each pixel in the set U to obtain a disparity value thereof, and performing optimization processing … After obtaining all pixel coordinates of the fish in the two-dimensional image, stereo matching is performed by using the binocular camera and the SGBM algorithm”). Notes: a set is necessarily a subset of itself); and
identify a second subset of one or more key points from the second set of key points (Figure 1 illustrates key point P’ as perceived via the left camera, and P’’ as perceived by the right camera; Page 2: “Solving an edge extraction and communication area of the target box to obtain a set U of all pixels in the target box, where all pixels {x i, y i} in the set U are pixels constituting the fish… performing parallax matching on each pixel in the set U to obtain a disparity value thereof, and performing optimization processing … After obtaining all pixel coordinates of the fish in the two-dimensional image, stereo matching is performed by using the binocular camera and the SGBM algorithm”),
wherein a first key point included in the first subset is matched to a second key point included in the second subset (Page 2: “Solving an edge extraction and communication area of the target box to obtain a set U of all pixels in the target box, where all pixels {x i, y i} in the set U are pixels constituting the fish… performing parallax matching on each pixel in the set U to obtain a disparity value thereof, and performing optimization processing … After obtaining all pixel coordinates of the fish in the two-dimensional image, stereo matching is performed by using the binocular camera and the SGBM algorithm”. Notes: in parallax matching, corresponding pixels in two images from different perspectives are matched), and
the first 3D point maps to the first key point and the second key point (Page 2: “Solving an edge extraction and communication area of the target box to obtain a set U of all pixels in the target box, where all pixels {x i, y i} in the set U are pixels constituting the fish… performing parallax matching on each pixel in the set U to obtain a disparity value thereof, and performing optimization processing … After obtaining all pixel coordinates of the fish in the two-dimensional image, stereo matching is performed by using the binocular camera and the SGBM algorithm”; Page 2: “After the binocular camera parameter is acquired, the SBM is used to perform disparity matching calculation on all pixels of the above-mentioned fish pixel point set U to obtain a disparity value d thereof, and after obtaining the disparity value, all the pixels in the U are (x, y, i, d)”. Notes: disparity values are an indicator of depth, and hence after disparity matching, the resulting pixels in U are 3D points).
Claim 3, being similar in scope to Claim 18, is rejected under the same rationale.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 2 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Zhejiang University (CN 111640152 A) in view of Bourke (Converting dual fisheye images into a spherical (equirectangular) projection, 2016).
Regarding Claim 17, the apparatus of 16 is rejected over Zhejiang University.
Zhejiang University does not teach that generating the first image comprises capturing a first fisheye image using the first lens of the camera and converting the captured first fisheye image into the first image, generating the second image comprises capturing a second fisheye image using the second lens of the camera and converting the captured second fisheye image into the second image, and each of the first image and the second image is an equidistant image or an equirectangular image.
However, Bourke teaches generating a first image comprising capturing a first fisheye image using the first lens of the camera and converting the captured first fisheye image into the first image (Example: “The following example will illustrate the main features of the algorithm implementation. It will be based upon two separate fisheye images, each with an aperture of 210 degrees”),
generating the second image comprises capturing a second fisheye image using the second lens of the camera and converting the captured second fisheye image into the second image (Example: “The following example will illustrate the main features of the algorithm implementation. It will be based upon two separate fisheye images, each with an aperture of 210 degrees”), and
each of the first image and the second image is an equidistant image or an equirectangular image (Example: “The left fisheye above mapped into equirectangular space is shown below, it fills more than half the equirectangular image because the lens is 210 degrees”; Refer to image below the previous text; Example: “Repeating for the right hand fisheye above gives the following, noting that it generally covers the second half of the equirectangular image and in this case is continuous across the 0 to 360 agle wrap”; Refer to the image below the previous text).
Zhejiang University and Bourke are considered analogous in the art with regards to the use of images from different perspectives captured by different image sensors, and combining the images into a cohesive representation of a scene. A Common motivation in the art is to combine images with overlapping perspectives to form an image representative of a wider capture angle, as is evident in Bourke (Example: “Each fisheye is located in a different part of the image (sensor) plane. The resulting panorama after compensating correctly for these camera/fisheye errors is shown below”; Refer to the image below the text). One would be motivated to utilize fisheye images in scenarios where a wider capture is desired, such as for capturing an image for an entire space.
Therefore, it would have been obvious to a person having ordinary skill in the art to combine the apparatus for determining a dimension value of a 3D space of Zhejiang University with the use two fisheye images from two different cameras of Bourke; Doing so would yield the predictable result of establishing dimensions of a 3D space using fisheye images.
Claim 2, being similar in scope to Claim 17, is rejected under the same rationale.
Claims 4-6 and 19-21 are rejected under 35 U.S.C. 103 as being unpatentable over Zhejiang University (CN 111640152 A) in view of Wu (US 20190122379 A1).
Regarding Claim 19, the apparatus of Claim 16 is rejected over Zhejiang University.
Zhejiang University does not teach a first directional vector having a first direction, wherein the first direction is from a first reference point of the first lens of the camera to one key point included in the first set of key points; and determining a second directional vector having a second direction, wherein the second direction from a second reference point of the second lens of the camera to one key point included in the second of key points.
However, Wu teaches a first directional vector having a first direction (Paragraph [0049]: “To be specific, FIG. 4 illustrates a schematic diagram of calculating three-dimensional coordinates of an object according to an embodiment of the invention. With reference to FIG. 4, it is assumed that O.sub.l and O.sub.r are the fisheye centers of the left and the right fisheye lenses, a point P is a projection point of the object T on a lens plane, and a length of a baseline O.sub.lO.sub.r is B”; Figure 4 illustrates a vector from 0_l to key point P. Notes: as mentioned previously, P is the projection of object T on a lens plane, and is hence established to be a key point),
wherein the first direction is from a first reference point of the first lens of the camera to one key point included in the first set of key points (Figure 4 illustrates a vector from 0_l to key point P. Notes: as mentioned previously, P is the projection of object T on a lens plane, and is hence established to be a key point); and
determining a second directional vector having a second direction (Paragraph [0049]: “To be specific, FIG. 4 illustrates a schematic diagram of calculating three-dimensional coordinates of an object according to an embodiment of the invention. With reference to FIG. 4, it is assumed that O.sub.l and O.sub.r are the fisheye centers of the left and the right fisheye lenses, a point P is a projection point of the object T on a lens plane, and a length of a baseline O.sub.lO.sub.r is B”; Figure 4 illustrates a vector from 0_r to key point P. Notes: as mentioned previously, P is the projection of object T on a lens plane, and is hence established to be a key point),
wherein the second direction from a second reference point of the second lens of the camera to one key point included in the second of key points (Figure 4 illustrates a vector from 0_r to key point P. Notes: as mentioned previously, P is the projection of object T on a lens plane, and is hence established to be a key point).
Zhejiang University and Wu are considered analogous in the art with respect to the use of two camera lenses for determining a distance from the camera to an object point. One would be motivated to obtain vectors from the camera lens positions to the object point for triangulation purposes, including determining an intersection point and an associated distance to the intersection point from the baseline containing the cameras.
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine the apparatus for determining a dimension value of Zhejiang University with the determination of a first and second vector from a first and second lens of a camera to a key point of Wu; Doing so would yield the predictable result of allowing for the determination of a distance from a center point representative of the camera as a whole to a key point via triangulation.
Claim 4, being similar in scope to Claim 19, is rejected under the same rationale.
Regarding Claim 20, the apparatus of Claim 19 is rejected over Zhejiang University as modified.
Zhejiang as modified teaches the apparatus of Claim 19, wherein the apparatus is further configured to:
perform a triangulation process using the first directional vector, the second directional vector, and a baseline between the first and second reference points, thereby determining an intersection point of the first directional vector and the second directional vector, wherein the real-world point is the intersection point (Wu, Paragraph [0045]: “In addition, the processor 18 further executes the incident angle calculation module 164 to transform a first distance between the first coordinates and the fisheye center of the first fisheye image and a second distance between the second coordinates and the fisheye center of the second fisheye image into a first incident angle and a second incident angle through lens curves of the first fisheye lens 12 and the second fisheye lens 14 (step S208)”; Wu, Paragraph [0048]: “After the azimuth angle of the object relative to the fisheye center and an incident angle calculated through a distance between the object and the fisheye center, the processor 18 executes the coordinate calculation module 165, so as to calculate three-dimensional coordinates of the object by using a trigonometric function according to the first azimuth angle, the second azimuth angle, the first incident angle, the second incident angle, and the baseline distance (step S210)”).
Claim 5, being similar in scope to Claim 20, is rejected under the same rationale.
Regarding Claim 21, the apparatus of Claim 16 is rejected over Zhejiang University.
Zhejiang University does not teach calculating a second distance value indicating a distance between the real-world point and the first lens of the camera, and does not teach calculating a third distance value indicating a distance between the real-world point and the second lens of the camera, wherein the first distance value is calculated using the second distance value and the third distance value.
However, Wu teaches calculating a second distance value indicating a distance between the real-world point and the first lens of the camera, and calculating a third distance value indicating a distance between the real-world point and the second lens of the camera (Paragraph [0007]: “In the method, a first fisheye image and a second fisheye image containing an object respectively are captured by using the first fisheye lens and the second fisheye lens. Next, first coordinates of the object in the first fisheye image and second coordinates of the object in the second fisheye image are detected. A first azimuth angle and a second azimuth angle of the object relative to fisheye centers of the first fisheye image and the second fisheye image on an image sensor plane of the first fisheye lens and the second fisheye lens are then calculated according to the first coordinates and the second coordinates. A first distance between the first coordinates and the fisheye center of the first fisheye image and a second distance between the second coordinates and the fisheye center of the second fisheye image are transformed into a first incident angle and a second incident angle respectively through lens curves of the first fisheye lens and the second fisheye lens”. Notes: Given that the coordinates for the fisheye lens and the object coordinates for each image are known, the distance to the coordinates from the first lens and the second lens are necessarily calculated from them),
Zhejiang University and Wu are considered analogous in the art with respect to the use of two cameras for determining a distance from the camera to an object point. One would be motivated to obtain vectors from the camera lens positions to the object point for use in triangulation to determine an intersection point of the vectors from the camera lenses, as is evident in Wu (Paragraph [0045]: “In addition, the processor 18 further executes the incident angle calculation module 164 to transform a first distance between the first coordinates and the fisheye center of the first fisheye image and a second distance between the second coordinates and the fisheye center of the second fisheye image into a first incident angle and a second incident angle through lens curves of the first fisheye lens 12 and the second fisheye lens 14 (step S208)”; Paragraph [0048]: “After the azimuth angle of the object relative to the fisheye center and an incident angle calculated through a distance between the object and the fisheye center, the processor 18 executes the coordinate calculation module 165, so as to calculate three-dimensional coordinates of the object by using a trigonometric function according to the first azimuth angle, the second azimuth angle, the first incident angle, the second incident angle, and the baseline distance (step S210)”).
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine the apparatus for determining a dimension value of Zhejiang University with the determination of a first and second distance from a first and second lens of a camera to a key point of Wu; Doing so would yield the predictable result of allowing for the determination of a first distance from a center point between the first lens and the second lens to a key point via triangulation.
Zhejiang University as modified does not explicitly teach that the first distance value is calculated using the second distance value and the third distance value.
However, Zhejiang University as modified teaches that the intersect of the two vectors representing a key point is calculated using the second distance value and the third distance value (Wu, Paragraph [0007]: “Finally, three-dimensional coordinates of the object are calculated by using a trigonometric function according to the first azimuth angle, the second azimuth angle, the first incident angle, the second incident angle, and the baseline distance”; Wu, Paragraph [0045]: “In addition, the processor 18 further executes the incident angle calculation module 164 to transform a first distance between the first coordinates and the fisheye center of the first fisheye image and a second distance between the second coordinates and the fisheye center of the second fisheye image into a first incident angle and a second incident angle through lens curves of the first fisheye lens 12 and the second fisheye lens 14 (step S208)”; Wu, Paragraph [0048]: “After the azimuth angle of the object relative to the fisheye center and an incident angle calculated through a distance between the object and the fisheye center, the processor 18 executes the coordinate calculation module 165, so as to calculate three-dimensional coordinates of the object by using a trigonometric function according to the first azimuth angle, the second azimuth angle, the first incident angle, the second incident angle, and the baseline distance (step S210)”). Considering that Figure 4 establishes a line from the baseline B between the two camera lenses to a key point T representing the real-world point, it would have been simple and obvious for a person to calculate the first distance value using the second distance value and the third distance value, where the point between the lenses is representative of the camera as a whole instead of from a particular lens on the camera.
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to modify the apparatus of Zhejiang University as modified to calculate the distance from a point representative of the camera as a whole to a point representative of a real-world point; Doing so would yield the predictable result of having a general distance from the camera itself to the real-world point.
Claim 6, being similar in scope to Claim 21, is rejected under the same rationale.
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Zhejiang University (CN 111640152 A) in view of Hahne (Baseline and Triangulation Geometry in a Standard Plenoptic Camera, 2017)
Regarding Claim 7, the method of Claim 1 is rejected over Zhejiang University.
Zhejiang University does not teach that the distance between the real-world point and the camera is a distance between the real- world point and a reference point in the camera, and the reference point is located between a location of the first lens and a location of the second lens.
However, Hahne teaches that the distance between the real-world point and the camera is a distance between the real- world point and a reference point in the camera, and the reference point is located between a location of the first lens and a location of the second lens (Section 2.1, Coplanar Stereo Cameras: “An object point’s depth distance Z can be directly fetched from parameters in Fig. 2. As highlighted with a dark tone of grey, [change in x] may represent the base of any acute scalene triangle with b as its height. Another triangle spanned by the base B and height Z is a scaled version of it and shown in light grey. This relationship relies on the method of similar triangles and can be written as an equality of ratios… To infer the depth distance Z, [Eq. 1] may be rearranged to [Eq. 2]”; Figure 2 illustrates an object distance Z from Baseline B between two cameras (lenses). Equation 1 and Equation 2 teach determining a distance Z from a baseline in which lenses lie upon).
Zhejiang University and Hahne are considered analogous in the art with respect to the determination of a distance between a real-world point and a camera. A common motivation for representing the distance between a camera with two lenses and a target point as the distance between a point between the lenses and the target point is to generalize the distance or depth of the point to the camera itself rather than to one of the lenses.
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine the method for determining a dimension value of Zhejiang University with the determination of a distance between the real-world point and the camera being a distance between the real-world point and a point between the first and second lens of Hahne; Doing so would yield the predictable result of a generalized distance from the camera, represented by a point between the two lenses, to the real-world point of interest.
Claim(s) 8-12 are rejected under 35 U.S.C. 103 as being unpatentable over Zhejiang University (CN 111640152 A) in view of Math Stack Exchange (Coordinate system transformation: from world coordinates to camera coordinates?, 2016).
Regarding Claim 8, the method of Claim 1 is rejected over Zhejiang University.
Zhejiang University does not teach converting original coordinates of said one or more 3D points into converted coordinates, wherein the original coordinates are in a first coordinate system, the converted coordinates are in a second coordinate system, a center of the first coordinate system is not a reference point of the camera, and a center of the second coordinate system is the reference point of the camera.
However, Math Stack Exchange teaches converting original coordinates of said one or more 3D points into converted coordinates, wherein the original coordinates are in a first coordinate system, the converted coordinates are in a second coordinate system, a center of the first coordinate system is not a reference point of the camera, and a center of the second coordinate system is the reference point of the camera (Answer by pbierre: “Coordinate translate: World to Eye. First, all world points must be put into local viewing coordinates where the camera defines the origin. This is done as a coordinate translation, where the camera's position coordinates are subtracted from each world point's coordinates....the result now is that all points revolve around an origin at the camera”).
Zhejiang University and Math Stack Exchange are considered analogous in the art with respect to the use of a coordinate system with respect to a camera. One would be motivated to convert coordinates to a second coordinate system centered around the camera such that the coordinate system is local to the camera, which would make coordinates within the system more intuitive with respect to where the camera is located.
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date to combine the method for determining a dimension value of Zhejiang University with the converted coordinates in the coordinate system centered around the camera of Math Stack Exchange; Doing so would yield the predictable result of the coordinates of Zhejiang being more intuitive with respect to where the camera of Zhejiang is located.
Regarding Claim 9, the method of Claim 8 is rejected over Zhejiang University as modified.
Zhejiang University as modified teaches the method of claim 8, wherein the converted coordinates of said one or more 3D points include a first converted coordinate of the first 3D point (Math Stack Exchange, Answer by pbierre: “Coordinate translate: World to Eye. First, all world points must be put into local viewing coordinates where the camera defines the origin. This is done as a coordinate translation, where the camera's position coordinates are subtracted from each world point's coordinates....the result now is that all points revolve around an origin at the camera”; ), and the method further comprises calculating a distance value between the reference point of the camera and the first converted coordinate of the first 3D point (Zhejiang University, Page 2: “After the binocular camera parameter is acquired, the SBM is used to perform disparity matching calculation on all pixels of the above-mentioned fish pixel point set U to obtain a disparity value d thereof, and after obtaining the disparity value, all the pixels in the U are (x, y, i, d); Zhejiang University, Figure 2; Zhejiang University, Page 2: “Finally, by using the camera parameters and its imaging principle, the distance (depth) depth from the [fish] to the camera is calculated according to the disparity value d of each pixel point, the principle is as shown in FIG. 2, x_left and x_right are coordinates of the real point at the left eye and the right eye camera, B (Baseline) is the camera baseline parameter, O_left and O_right are dual gaze centers, and f is the focal length. The depth may be solved by a formula depth = f * B/(x_left-x_right)”; Zhejiang University, Page 2: “After the depth of each pixel is obtained, the average distance D from all the pixels in the set U to the camera is obtained by averaging, and the average distance D is used as the distance from the fish to the camera”. Notes: each depth value corresponds with a distance from the camera to a real-world point corresponding to a pixel point in U).
Regarding Claim 10, the method of Claim 9 is rejected over Zhejiang University.
Zhejiang University as modified teaches determining a scaling factor based on a ratio of a distance value and a reference distance value, wherein the dimension value is determined based on the scaling factor value (Zhejiang University, Page 2: “The length of all the target fish in the image may be solved through the average distance D, the approximate length w of the fish in the picture, and the similar triangular principle. The estimation principle is shown in FIG. 3, L is the actual size of the fish, (u_0, v_1), (u_0, v_0), (u_1, v_1), (u_1, v_0) is the pixel coordinates of the fish in the camera, w = (u_1 − u_0), or in the U set, w = (x_max − x_min), x_max is the maximum horizontal coordinate in U, and x_min is the minimum horizontal coordinate. It is known that Z is the average depth D, 0 is the camera optical center, f is the focal length, and the length L of the target fish can be solved by the formula L = w(Z-f)/f”. Notes: w is the width of the fish, Z is the depth avg (distance from object points to camera average), and f is the focal length of the camera. In this case, the ratio term is w(Z-f)/f, which is used as a scaling factor for obtaining a dimension value L from a reference value w and distance value Z).
Zhejiang University as modified does not explicitly teach that the reference distance value is between the camera and the first converted coordinate of the first 3D point.
However, a person having ordinary skill in the art would appreciate that the reference distance value w can be any length in the space in which a dimension value is to be calculated for, which includes a reference distance value indicating a distance between the reference point of the camera and the first converted coordinate of the first 3D point. One would be motivated to do so to obtain a dimension representing the distance a camera to a point of interest.
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to modify the reference distance value of Zhejiang University as modified to indicate a distance between the reference point of the camera and the first converted coordinate of the first 3D point; Doing so would yield the predictable result of obtaining a dimension value pertaining to the distance between the reference point of the camera and the first converted coordinate of the first 3D point.
Regarding Claim 11, the method of Claim 10 is rejected over Zhejiang as modified.
Zhejiang as modified teaches the method of Claim 10, wherein the method further comprises:
Calculating a distance value indicating a distance between the reference point of the camera and a real world point mapped to each of the original coordinates of said one or more 3D points (Zhejiang University, Page 2: “After the binocular camera parameter is acquired, the SBM is used to perform disparity matching calculation on all pixels of the above-mentioned fish pixel point set U to obtain a disparity value d thereof, and after obtaining the disparity value, all the pixels in the U are (x, y, i, d); Zhejiang University, Figure 2; Zhejiang University, Page 2: “Finally, by using the camera parameters and its imaging principle, the distance (depth) depth from the [fish] to the camera is calculated according to the disparity value d of each pixel point, the principle is as shown in FIG. 2, x_left and x_right are coordinates of the real point at the left eye and the right eye camera, B (Baseline) is the camera baseline parameter, O_left and O_right are dual gaze centers, and f is the focal length. The depth may be solved by a formula depth = f * B/(x_left-x_right)”; Zhejiang University, Page 2: “After the depth of each pixel is obtained, the average distance D from all the pixels in the set U to the camera is obtained by averaging, and the average distance D is used as the distance from the fish to the camera”. Notes: each depth value corresponds with a distance from the camera to a real-world point corresponding to a pixel point in U);
Calculating a reference distance value indicating a distance between the reference point of the camera and each of the converted coordinates of said one or more 3D points (Zhejiang University, Page 2: “The length of all the target fish in the image may be solved through the average distance D, the approximate length w of the fish in the picture, and the similar triangular principle”; Refer to the argument for obviousness for a distance between the reference point of the camera and a converted coordinate from the rejection of Claim 10);
Determining, for each of said one or more 3D points, a scaling factor value based on a ratio of the distance value and reference distance value (Zhejiang University, Page 2: “The length of all the target fish in the image may be solved through the average distance D, the approximate length w of the fish in the picture, and the similar triangular principle. The estimation principle is shown in FIG. 3, L is the actual size of the fish, (u_0, v_1), (u_0, v_0), (u_1, v_1), (u_1, v_0) is the pixel coordinates of the fish in the camera, w = (u_1 − u_0), or in the U set, w = (x_max − x_min), x_max is the maximum horizontal coordinate in U, and x_min is the minimum horizontal coordinate. It is known that Z is the average depth D, 0 is the camera optical center, f is the focal length, and the length L of the target fish can be solved by the formula L = w(Z-f)/f”. Notes: w is the width of the fish, Z is the depth avg (distance from object points to camera average), and f is the focal length of the camera. In this case, the ratio term is w(Z-f)/f, which is used for obtaining a dimension value L from a reference value w and distance value Z); and
Calculating an average of the scaling factors of said one or more 3D points (Zhejiang University, Page 2: “The length of all the target fish in the image may be solved through the average distance D, the approximate length w of the fish in the picture, and the similar triangular principle. The estimation principle is shown in FIG. 3, L is the actual size of the fish, (u_0, v_1), (u_0, v_0), (u_1, v_1), (u_1, v_0) is the pixel coordinates of the fish in the camera, w = (u_1 − u_0), or in the U set, w = (x_max − x_min), x_max is the maximum horizontal coordinate in U, and x_min is the minimum horizontal coordinate. It is known that Z is the average depth D, 0 is the camera optical center, f is the focal length, and the length L of the target fish can be solved by the formula L = w(Z-f)/f”. Notes: The equation w(Z-f)/f of Zhejiang, in its equivalent expanded form, is the average of w(x-f)/f for all x in X, where Z is the average of all x in X; this follows the principle that averaging operations commute with linear operations. X is representative of the set U, which defines the set of 3D points).
Regarding Claim 12, the method of Claim 11 is rejected over Zhejiang University as modified.
Zhejiang as modified teaches the method of Claim 11, wherein the dimension value is determined based on the average of the scaling factors (Zhejiang University, Page 2: “The length of all the target fish in the image may be solved through the average distance D, the approximate length w of the fish in the picture, and the similar triangular principle. The estimation principle is shown in FIG. 3, L is the actual size of the fish, (u_0, v_1), (u_0, v_0), (u_1, v_1), (u_1, v_0) is the pixel coordinates of the fish in the camera, w = (u_1 − u_0), or in the U set, w = (x_max − x_min), x_max is the maximum horizontal coordinate in U, and x_min is the minimum horizontal coordinate. It is known that Z is the average depth D, 0 is the camera optical center, f is the focal length, and the length L of the target fish can be solved by the formula L = w(Z-f)/f”. Notes: The equation w(Z-f)/f of Zhejiang, in its equivalent expanded form, is the average of w(x-f)/f for all x in X, where Z is the average of all x in X; this follows the principle that averaging operations commute with linear operations. X is representative of the set U, which defines the set of 3D points. L is the dimension value of the length of interest corresponding with the reference distance value. The ratio term is w(Z-f)/f, which is used as a scaling factor for obtaining a dimension value L from a reference value w and distance value Z).
Claim 13 is rejected under 35 U.S.C. 103 as being unpatentable over Zhejiang University (CN 111640152 A) in view of El Dokor (US 20150332474 A1).
Regarding Claim 13, the method of Claim 1 is rejected over Zhejiang University.
Zhejiang University does not teach displaying at least a part of the 3D space with an indicator indicating the physical dimension.
However, El Dokor teaches displaying at least a part of the 3D space with an indicator indicating the physical dimension (Figure 12 clearly demonstrates displaying an indicator of the physical dimension with a part of the 3D space).
Zhejiang University and El Dokor are considered analogous in the art with respect to determining dimension values with respect to a 3D space captured by a camera. One would be motivated to display an indicator of the physical dimension along with the 3D scene to improve the viewing experience, as is evident in El Dokor.
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine the method of Zhejiang University with the display of an indicator indicating the physical dimension with a part of the 3D space of El Dokor; Doing so would yield the predictable result of improving the viewer experience and providing more context for the 3D scene with respect to dimension values.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to RAYMOND CHUN LAM LI whose telephone number is (571)272-5124. The examiner can normally be reached M-F 8:30-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kent Chang can be reached at 571-272-7667. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RAYMOND CHUN LAM LI/Examiner, Art Unit 2614
/KENT W CHANG/Supervisory Patent Examiner, Art Unit 2614
Prosecution Insights