DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Specification
The disclosure is objected to because of the following informalities: paragraphs 13, 22, 93 and 97 “possion” should read “poisson”.
Appropriate correction is required.
Claim Objections
Claim 25, 27-28, 30, 32, and 34 objected to because of the following informalities: typos.
Claim 25: “the product” should read “a product”.
Claim 27, last line, “of at least one pixel” should read “of the at least one pixel”.
Claim 28, each instance of, “of at least one pixel” should read “of the at least one pixel”.
Claims 30, 32, and 34, “possion” should read “poisson”.
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 22-41 rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 22 and 39-41 recites the limitation "a scene space model" in first line and second to last line (for claim 22) and last line (for each of claims 39-41). There is insufficient antecedent basis for this limitation in the claim. This is because it is unclear whether the two scene space model mentions are the same instance or a different instance of scene space model.
Claim 25 recites the limitation "the point cloud matching algorithm" in last 3 lines. There is insufficient antecedent basis for this limitation in the claim. This is because it’s unclear what point cloud matching algorithm is being referred to since there is no previous mention in the parent claims of point cloud matching algorithm.
Claim 25 recites the limitation "the third fixed-point rotation matrix" in lines 5 and 8. There is insufficient antecedent basis for this limitation in the claim. This is because it’s unclear what third fixed-point rotation matrix is being referred to since there is no previous mention in the parent claims of third fixed-point rotation matrix.
Claim 26 recites the limitation "a three-dimensional unit sphere" in second and last line. There is insufficient antecedent basis for this limitation in the claim. This is because it’s unclear what three-dimensional unit sphere is being referred to since there is another instance of “a three-dimensional unit sphere” mentioned in claim 22 making it unclear if this is the same instance as that instance or different.
Claim 27 and 28 recites the limitation "a second panoramic image" in lines 2 and 7 (for claim 27) and lines 1-2 (for claim 28). There is insufficient antecedent basis for this limitation in the claim. This is because it’s unclear what second panoramic image is being referred to since there is another instance of “a second panoramic image” mentioned in the parent claim(s) making it unclear if this is the same instance as that instance or a different instance.
Claims 29, 31, 33, 35, 36, 37, and 38 recites the limitation "the mesh…corresponding mesh" in lines 7 and 9 (for claim 29, 31, 33, 35, 36, 37, and 38). There is insufficient antecedent basis for this limitation in the claim. This is because it’s unclear what mesh is being referred to since there is no previous mention in the parent claims of a mesh and the claim itself mention “a mesh model” which is not interpreted to be the same as “the mesh” since “the mesh” is “of the mesh model”. Also, “corresponding mesh” further adds to the confusion due to no previous mention of “a mesh”.
Claims 30, 32, and 34 recites the limitation "the mesh " in line 2 of each claim. There is insufficient antecedent basis for this limitation in the claim. This is because it’s unclear what mesh is being referred to since there is no previous mention in the parent claims of “a mesh”.
Claims 23-24 rejected under 35 U.S.C. 112(b) since they depend on a claim that is rejected under rejected under 35 U.S.C. 112(b).
Note. Most likely these claims depend on some dependent claim or are missing elements.
In order to fix this issue, dependency should be reviewed and any first instance of an element
should be made clear that it’s a first instance and should be referred to as “a” or “an” instead of
“the”, and if multiple instances exist, further instances should be further distinguished for example by saying “first”, “second”, and/or “third” etc.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 39, recite a computer-readable storage medium. The broadest reasonable
interpretation of a claim drawn to a computer readable medium (also called machine readable
medium and other such variations) typically covers forms of non-transitory tangible media and
transitory propagating signals per se in view of the ordinary and customary meaning of computer
readable media, particularly when the specification is silent. See MPEP 2111.01. When the
broadest reasonable interpretation of a claim covers a signal per se, the claim must be rejected
under 35 U.S.C. 101 as covering non-statutory subject matter. The USPTO recognizes that
applicants may have claims directed to computer readable media that cover signals per se, which
the USPTO must reject under 35 U.S.C. 101 as covering both non-statutory subject matter and
statutory subject matter. A claim drawn to such a computer readable medium that covers both
transitory and non-transitory embodiments may be amended to narrow the claim to cover only
statutory embodiments to avoid a rejection under 35 U.S.C. 101 by adding the limitation "non-
transitory" to the claim. Such an amendment would typically not raise the issue of new matter,
even when the specification is silent because the broadest reasonable interpretation relies on the
ordinary and customary meaning that includes signals per se.
Applicant’s specification in paragraph 118 recites “The computer-readable storage medium may be any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium.” Since Applicant’s disclosure does not limit the definition of “computer-readable storage medium”, it could be a signal.
As an additional note, a non-transitory computer readable medium having executable
programming instructions stored thereon is considered statutory as non-transitory computer
readable media excludes transitory data signals.
Claim 41 is rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. The claim(s) does/do not fall within at least one of the four categories of patent eligible subject matter because the claim 41 recites: “A computer program product, including computer program instructions, wherein the computer program instructions, when executed by a processor, perform the method for constructing a scene space model of claim 22.”. The body of the claim recites computer program steps, such as,“constructing a scene…”, which are nothing more than just programmed instructions to be performed by
the system. Similarly, computer programs claimed as computer listings per se, ie., the
descriptions or expressions of the programs, are not physical “things.” They are neither computer
components nor statutory processes, as they are not “acts” being performed. Such claimed
computer programs do not define any structural and functional interrelationships between the
computer program and other claimed elements of a computer which permit the computer
program’s functionality to be realized. In contrast, a claimed non-transitory computer-readable
medium encoded with a computer program is a computer element which defines structural and
functional interrelationships between the computer program and the rest of the computer which
permit the computer program’s functionality to be realized, and is thus statutory. Accordingly, it
is important to distinguish claims that define descriptive material per se from claims that define
statutory inventions. Applicant’s specification paragraph 112 mentions “The computer program products may include various forms of computer-readable storage media, such as volatile memory and/or non-volatile memory. Volatile memory, for example, may include: random access memory (RAM) and/or cache memory (cache), etc. Non-volatile memory, for example, may include: read-only memory (ROM), hard disk, flash memory, etc”, however, this does not constrain the computer program product to being encoded in hardware in the claim language itself.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 22, 27-28 and 39-41 is/are rejected under 35 U.S.C. 103 as being unpatentable over Aparicio Ojea et al. (U.S. Patent Application Publication No. 2022/0410391), hereinafter referenced as Ojea, ZHANG et al. (U.S. Patent Application Publication No. 2023/0103385), hereinafter referenced as Zhang and Zeng et al. (The algorithm to generate color point-cloud with the registration between panoramic image and laser point-cloud), hereinafter referenced as Zeng.
Regarding claim 22, Ojea teaches A method for constructing a scene space model, comprising: (paragraph 22 teaches "FIG. 1, an example industrial or physical environment or scene 100 is shown. As used herein, the industrial environment 100 can refer to a physical environment, and a simulated or simulation environment (or world model) can define a virtual representation of the physical environment"); simulation environment / world model defining virtual representation of physical environment shows constructing a scene space model; acquiring first point cloud information corresponding to a target scene collected by mobile point cloud collection equipment (paragraph 29 teaches "robot device 104 can further include one or more cameras, for instance an imaging sensor 212 mounted on the arm 106 of the robot device 104. The imaging sensor 212 can be configured to generate a 3D point cloud of a given scene"); camera mounted to arm (mobile/moves) which generates point cloud shows first point cloud information would be collected using mobile point cloud collection equipment and the given scene that is being imaged is the target scene; acquiring depth image information corresponding to a partial region of the target scene collected by fixed-point depth camera equipment, (paragraph 23 teaches "The system 102 can further include one or more cameras or sensors, for instance a three-dimensional (3D) point cloud camera 112, configured to detect and record objects in the environment 102. The camera 112 can be configured to generate a 3D point cloud of a given scene, for instance the environment 100....record images...depth images"); this shows depth image being recorded, of objects in environment shows it is for partial region of target scene and since this is from camera 112 which is shown to be affixed in fig. 1, the information collected is by a fixed-point depth camera equipment; wherein the depth image information includes: second point cloud information and image color information corresponding to the second point cloud information (paragraph 23 teaches "The camera 112 can be configured to generate a 3D point cloud of a given scene, for instance the environment 100. Alternatively, or additionally, the one or more cameras of the system 102 can include one or more standard two-dimensional (2D) cameras that can record images (e.g., RGB images or depth images) from different viewpoints"); RGB or depth images shows the 2D cameras (multiple thus each) would record RGB and depth images respectively thus showing the point cloud information (second point cloud information) generated from camera 112 has image RGB/color information corresponding to it as well, therefore the depth image corresponds to and includes both color and second point cloud information;
However, Ojea fails to teach determining a rotation matrix of a camera coordinate system of the fixed- point depth camera equipment corresponding to a global coordinate system based on the first point cloud information and the second point cloud information, wherein the global coordinate system is a coordinate system corresponding to the first point cloud information; generating a first panoramic image based on the depth image information, mapping the first panoramic image onto a three-dimensional unit sphere; rotating the three-dimensional unit sphere based on the rotation matrix to generate a second panoramic image; and generating a scene space model based on the first point cloud information and the second panoramic image.
However, Zhang teaches determining a rotation matrix of a camera coordinate system of the fixed- point depth camera equipment corresponding to a global coordinate system based on the first point cloud information and the second point cloud information, (Zhang, claim 5 teaches "unifying the point clouds under the world coordinate system to obtain a plurality of point clouds under different viewing angles comprises: under each viewing angle k, defining a translation vector…and defining a rotation matrix...transformation between the camera coordinate system and the world coordinate system being done as follows:"); this shows defining/determining rotation matrix that leads to transformation of camera coordinate system (of fixed-point depth camera from Ojea when viewed in combination) and it is corresponding to world/global coordinate system as well as based on the two point cloud information’s since unifies point clouds; wherein the global coordinate system is a coordinate system corresponding to the first point cloud information (Zhang, paragraph 57 teaches "The point clouds under the same viewing angle, i.e., under the same camera, are located in the corresponding camera coordinate system. The purpose of external parameter calibration is to transform the point clouds under different camera coordinate systems to the same world coordinate system to complete point cloud stitching."); transforming point clouds (inclusive of first) to same world/global coordinate system shows global coordinate system initially corresponding to first point cloud information. Zhang is considered to be analogous art because it is reasonably pertinent to the problem faced by the inventor of coordinate systems alongside camera and 3D reconstruction. Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination of Ojea and Zhang invention with the coordinates and reconstruction techniques of Zhang to for convergence and higher accuracy, and the method is simple in process and easy to operate and implement (Zhang, paragraph 34). This is done by having the unified point clouds under world coordinate system.
However, the combination of Ojea and Zhang fails to teach generating a first panoramic image based on the depth image information, mapping the first panoramic image onto a three-dimensional unit sphere; rotating the three-dimensional unit sphere based on the rotation matrix to generate a second panoramic image; and generating a scene space model based on the first point cloud information and the second panoramic image.
However, Zeng teaches generating a first panoramic image based on the depth image information, (Zeng, page 5, first two paragraphs teaches "Panoramic images are collected every 5... select the nearest panoramic image for each point based on GPS time or geometric distance"); panoramic image collected then selected means it must be generated and doing this based on geometric distance shows being based on the depth image information; mapping the first panoramic image onto a three-dimensional unit sphere (Zeng, page 3, first paragraph teaches "Local three-dimensional Cartesian coordinate system S1(X1Y1Z1): The system origin is in the center of the current panoramic sphere." and fig. 4 shows panoramic sphere); originating in center shows mapping beginning there and fig. 4 shows result of panoramic image mapped onto 3D unit sphere; rotating the three-dimensional unit sphere based on the rotation matrix to generate a second panoramic image (Zeng, page 4, step 2 and equation 2 teaches "Step2. Coordinates transform form S1 (X1Y1Z1) to S2(XsYsZs), as Formula 2...Where, (a1,a2,a3,b1,b2,b3,c1,c2,c3) are the parameters of the rotation matrix, determined by
the three attitude angles of the panoramic sphere:"); this shows applying rotation matrix to panoramic sphere/3D unit sphere and the result (at end of steps) is the generated second panoramic image; and generating a scene space model based on the first point cloud information and the second panoramic image (Zeng, fig. 7 shows color point cloud depicting a scene space as a model and abstract teaches "we introduce a method to generate color point-cloud with panoramic image and laser point-cloud,"); this comes after having first point cloud (laser point cloud) and the second panoramic image, thus based on such. Zeng is considered to be analogous art because it is reasonably pertinent to the problem faced by the inventor of generation of panoramic images and spheres using rotation matrix. Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination of Ojea and Zhang with the panoramic and sphere techniques of Zeng to ensure series of data fusion process, the laser point-cloud and images can registration accurately (Zeng, page 2, paragraph 2). This would be done by the specific computations.
Regarding claim 27, the combination of Ojea, Zhang, and Zeng teaches wherein the rotating the three- dimensional unit sphere based on the rotation matrix to generate a second panoramic image includes: rotating the three-dimensional unit sphere based on the rotation matrix to acquire new three-dimensional coordinates of at least one pixel of the first panoramic image (Zeng, page 4, step 2 and equation 2 teaches "Step2. Coordinates transform form S1 (X1Y1Z1) to S2(XsYsZs), as Formula 2...Where, (a1,a2,a3,b1,b2,b3,c1,c2,c3) are the parameters of the rotation matrix, determined by
the three attitude angles of the panoramic sphere:", and step 3 teaches "Calculate the polar coordinates P(B,L,R)of the image point in the panoramic sphere by the coordinates (X Y Z ) of the corresponding point in S2"); this shows applying rotation matrix to panoramic sphere/3D unit sphere and new 3D coordinates are obtained of image point/pixel(s) of panoramic image; and generating a second panoramic image based on the new three-dimensional coordinates and the color information of at least one pixel (Zeng, page, 4, step 6 and formula 8 teaches "Assign the object points with the RGB value of the pixels"); this comes in step after the aforementioned new 3D coordinates and shows color/RGB is applied to the pixel(s) (thus based on such) so that the resulting/final panoramic (second) image can be generated. The same motivations used in claim 22 apply here in claim 27.
Regarding claim 28, the combination of Ojea, Zhang, and Zeng teaches wherein the generating a second panoramic image based on the new three-dimensional coordinates and the color information of at least one pixel includes: determining a new position of at least one pixel of the first panoramic image on the three-dimensional unit sphere based on the new three-dimensional coordinates (Zeng, page 4, step 2 and equation 2 teaches "Step2. Coordinates transform form S1 (X1Y1Z1) to S2(XsYsZs), as Formula 2...Where, (a1,a2,a3,b1,b2,b3,c1,c2,c3) are the parameters of the rotation matrix, determined by
the three attitude angles of the panoramic sphere:", step 3 teaches "Calculate the polar coordinates P(B,L,R)of the image point in the panoramic sphere by the coordinates (X Y Z ) of the corresponding point in S2", and step 5 teaches Calculate the pixel’s coordinates (m, n) by the image plane coordinates"); new position here (of first panoramic image on 3D sphere) is determined by calculating pixel's coordinates and this comes in step after the aforementioned new 3D coordinates (thus based on such); and adding color information of at least one pixel of the first panoramic image to the new position to generate the second panoramic image (Zeng, page, 4, step 6 and formula 8 teaches "Assign the object points with the RGB value of the pixels...Where, RGB(Xs,Ys,Zs) means the RGB values of the point(Xs,Ys,Zs), N means the serial number of the images, RGB(m,n,N) means the RGB values of the pixel (m,n)."); assigning object points with RGB values of pixels shows adding color information of pixels (of the first panoramic image) to new position / object points so that the resulting/final panoramic (second) image can be generated. The same motivations used in claim 22 apply here in claim 28.
Regarding claim 39, the computer-readable storage medium claim 39 recites similar limitations as method claim 22, and thus is rejected under similar rationale. In addition, Ojea paragraph 48 teaches “a processor that executes machine-readable instructions stored on a computer readable medium, for performing tasks and may comprise any one or combination of, hardware and firmware”.
Regarding claim 40, the device claim 40 recites similar limitations as method claim 22, and thus is rejected under similar rationale. In addition, Ojea fig. 5 shows computing device 510 with memory 530 and processors 520. Ojea, paragraph 50 teaches “A processor may also comprise memory storing machine-readable instructions executable for performing tasks”.
Regarding claim 41, the computer program product claim 41 recites similar limitations as method claim 22, and thus is rejected under similar rationale. In addition, Ojea fig. 5 shows memory 530 and processors 520. Ojea, paragraph 50 teaches “A processor may also comprise memory storing machine-readable instructions executable for performing tasks” and paragraph 59 teaches “computer program products”.
Claim(s) 26 is/are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Ojea, Zhang and Zeng as applied to claims 22 above, and further in view of Li (U.S. Patent Application Publication No. 2014/0085295), hereinafter referenced as Li.
Regarding claim 26, the combination of Ojea, Zhang and Zeng fails to teach wherein the mapping the first panoramic image onto a three-dimensional unit sphere includes: converting at least one pixel in the first panoramic image from two- dimensional coordinates to three-dimensional coordinates to map the first panoramic image to a three-dimensional unit sphere.
However, Li teaches wherein the mapping the first panoramic image onto a three-dimensional unit sphere includes: converting at least one pixel in the first panoramic image from two- dimensional coordinates to three-dimensional coordinates to map the first panoramic image to a three-dimensional unit sphere (Li, abstract teaches "providing the panoramic image in a memory, the panoramic image being defined by a set of pixels in a 2-dimensional space; providing a model of the object, the model having a set of vertices in a 3-dimensional space" and claim 1 teaches "method for mapping a panoramic image to a 3-D virtual object of which a projection is made for display on a screen, comprising: providing the panoramic image in a memory, the panoramic image defined by a set of picture elements (pixels) in a 2-dimensional space; providing a model of the object, the model comprising a set of vertices in a 3-dimensional space"); the mapping of set of 2D pixels from panoramic image to 3D space shows conversion from 2D to 3D coordinates, thus transforming (first from Zeng) panoramic image to 3D unit (sphere from Zeng). Li is considered to be analogous art because it is reasonably pertinent to the problem faced by the inventor of mapping of 2D panoramic image pixels to 3D virtual object. Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination of Ojea, Zhang and Zeng invention with the 2D to 3D mapping techniques of Li for storing in memory an association between the selected vertex on the model and a value of the identified pixel (Li, abstract). This would help with directly relating the 2D to 3D, being able to access it later, and certain tasks like texture mapping and/or projecting changes.
Claim(s) 29-30 is/are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Ojea, Zhang and Zeng as applied to claim 22 above, and further in view of Adam et al. (U.S. Patent Application Publication No. 2024/0040101), hereinafter referenced as Adam.
Regarding claim 29, the combination of Ojea, Zhang and Zeng fails to teach wherein the generating the scene space model based on the first point cloud information and the second panoramic image includes: performing surface reconstruction processing on the first point cloud information based on a surface reconstruction algorithm to generate a mesh model corresponding to the first point cloud information; generating a texture of the mesh based on position information of the mesh of the mesh model and the second panoramic image; and setting the texture at corresponding mesh to generate a three-dimensional space model.
However, Adam teaches wherein the generating the scene space model based on the first point cloud information and the second panoramic image includes: performing surface reconstruction processing on the first point cloud information based on a surface reconstruction algorithm to generate a mesh model corresponding to the first point cloud information (Adam, paragraph 66 teaches "a mesh M is generated as illustrated by FIG. 1E from the point clouds obtained,...according to conventional methods, such as by reconstruction of Poisson surfaces "); reconstruction of Poisson surfaces shows surface reconstruction algorithm which generates the mesh that's from (corresponds to) the point cloud information (first point cloud information from Ojea when viewed in combination), thus the surface reconstruction processing of the first point cloud information is based on such; generating a texture of the mesh based on position information of the mesh of the mesh model and the second panoramic image (Adam, paragraph 66 teaches "a mesh M is generated as illustrated by FIG. 1E from the point clouds obtained, and a texture intended to be applied thereto "); since texture is applied to mesh, it must be based on position information and when viewed in combination this is also based on second panoramic image from Zeng since that image is constructed after rotation, whereas this is a reconstruction which would occur after initial construction; and setting the texture at corresponding mesh to generate a three-dimensional space model (Adam, claim 1 teaches "volumetric video stream of a three-dimensional action scene represented by a plurality of frames, the method being implemented by computer, and according to which a mesh and a texture for each frame of the scene are generated"); 3D action scene shows 3D space model being generated and having it be in accordance to mesh and texture shows texture would have to be set at the corresponding mesh. Adam is considered to be analogous art because it is reasonably pertinent to the problem faced by the inventor of surface reconstruction using mesh and texture. Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination of Ojea, Zhang and Zeng with the reconstruction, mesh and texture techniques of Adam to improve firstly the compression of the information relating to the meshes of the frames of a volumetric video (Adam, paragraph 27). This would create a more efficient system and would be done due to the techniques of the specific surface reconstruction.
Regarding claim 30, the combination of Ojea, Zhang, Zeng and Adam teaches wherein the surface reconstruction algorithm includes: Possion surface reconstruction algorithm (Adam, paragraph 66 teaches "such as by reconstruction of Poisson surfaces"); this shows Possion surface reconstruction algorithm; the mesh includes: a triangular mesh and a quadrilateral mesh (Adam, paragraph 7 teaches "as a basis for the modeling of the scene by a mesh of triangles connected continuously to one another, onto which a texture is applied" and paragraph 82 teaches "Block compression is a conventional compression method making it possible to reduce the quantity of memory required to store color data, wherein blocks of pixels (such as squares of 4 pixels by 4 pixels or 8 pixels by 8 pixels) are compressed"); this shows triangular mesh and quadrilateral mesh included in the mesh since one of ordinary skill in the art would understand that the block compression technique used for the texture to get square/quadrilateral blocks would also be used for the mesh in cases and places where the squares better fit the mesh in lieu of the triangle and lead to a more smoothened output. The same motivations used in claim 29 apply here in claim 30.
Claim(s) 23-25 is/are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Ojea, Zhang and Zeng as applied to claim 22 above, and further in view of Zhu et al. (U.S. Patent Application Publication No. 2020/0150275), hereinafter referenced as Zhu, and Meekins et al. (U.S. Patent Application Publication No. 2016/0350618), hereinafter referenced as Meekins.
Regarding claim 23, the combination of Ojea, Zhang and Zeng fails to teach wherein the determining the rotation matrix corresponding to the camera coordinate system of the fixed-point depth camera equipment and the global coordinate system based on the first point cloud information and the second point cloud information includes: splicing the second point cloud information collected by at least one fixed- point depth camera equipment to generate third point cloud information corresponding to a panoramic view of the target scene; determining a fixed-point rotation matrix of a camera coordinate system of the fixed-point depth camera equipment corresponding to the third point cloud information; determining a second fixed-point rotation matrix between the first point cloud information and the second point cloud information; and determining the rotation matrix based on the first fixed-point rotation matrix and the second fixed-point rotation matrix.
However, Zhu teaches wherein the determining the rotation matrix corresponding to the camera coordinate system of the fixed-point depth camera equipment and the global coordinate system based on the first point cloud information and the second point cloud information includes: splicing the second point cloud information collected by at least one fixed- point depth camera equipment to generate third point cloud information corresponding to a panoramic view of the target scene (Zhu, paragraph 25 teaches "scene's 3-D point cloud captured by sensor 140 with a 360-degree FOV, such as a LiDAR scanner" and paragraph 26 teaches "point cloud may be segmented into segments, each of which can be aggregated separately (for example in parallel) with corresponding images."); since fixed-point depth camera from above can be sensor and Ojea paragraph 21 mentions "world model can utilize cameras, lidars", the Lidar here is treated as the fixed-point depth camera equipment, thus the segmenting is done on the second point cloud information which leads to segmented/third point cloud that corresponds to 360-degree (panoramic) FOV of target scene from above; determining a fixed-point rotation matrix of a camera coordinate system of the fixed-point depth camera equipment corresponding to the third point cloud information (Zhu, paragraph 25 teaches "In some embodiments, to improve the accuracy of the color point cloud, the correlation between distances and calibration parameters (e.g., rotation matrices and translation vectors) may be taken into account in aggregating the point cloud and corresponding images" and paragraph 49 teaches "feature points in each point cloud group and the corresponding pixels in the associated images may be used for calculating the calibration parameters (e.g., rotation matrices and translation vectors) specific to the point cloud group."); rotation matrices specific to point cloud group shows a fixed-point rotation matrix corresponding to segmented/third point cloud information and this would be of camera coordinate system of a fixed-point depth camera when viewed in combination with the references above; Zhu is considered to be analogous art because it is reasonably pertinent to the problem faced by the inventor of arriving at multiple point clouds from splicing/segmenting. Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination of Ojea, Zhang and Zeng with the splicing point cloud techniques of Zhu to increase the processing speed (paragraph 25). This would be done by having smaller point clouds which can be processed in parallel or as needed leading to a more efficient system.
However, the combination of Ojea, Zhang, Zeng and Zhu fails to teach determining a second fixed-point rotation matrix between the first point cloud information and the second point cloud information; and determining the rotation matrix based on the first fixed-point rotation matrix and the second fixed-point rotation matrix.
However, Meekins teaches determining a second fixed-point rotation matrix between the first point cloud information and the second point cloud information (Meekins, paragraph 73 teaches "proper alignment can minimize the difference between two clouds of points...A cross product between the two sets of “matching” points results in a 3×3 matrix that is broken down with a Singular Value Decomposition. For example, the 3×3 matrix can be decomposed into three separate matrices U, S, and V"); second fixed-point rotation matrix (fixed-point since when viewed in combination, second point cloud is from fixed-point depth camera equipment) is shown here as V and is result of difference between two point clouds (first and second); and determining the rotation matrix based on the first fixed-point rotation matrix and the second fixed-point rotation matrix (Meekins, paragraph 73 teaches "The U and V matrices can be multiplied, ignoring the S matrix, and a resulting rotation matrix can be used to move the point cloud towards the target points. This resulting matrix can be applied to all points to be rotated"); to achieve a rotation matrix by multiplying matrices, two rotation matrices would have to be multiplied, thus the rotation matrix here is based on matrices U and V which are first and second fixed-point rotation matrices from above. Meekins is considered to be analogous art because it is reasonably pertinent to the problem faced by the inventor of matrices and computation of such regarding point clouds and rotations. Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination of Ojea, Zhang, Zeng and Zhu with the ICP and specific algorithmic computation techniques of Meekins to ensure a proper alignment can minimize the difference between two clouds of points (Meekins, paragraph 73). This would be done by the ICP process and precise matrices calculations.
Regarding claim 24, the combination of Ojea, Zhang, Zeng, Zhu and Meekins teaches wherein the determining the second fixed-point rotation matrix between the first point cloud information and the second point cloud information includes: matching the first point cloud information and the second point cloud information based on a preset point cloud matching algorithm to acquire the second fixed- point rotation matrix between the first point cloud information and the second point cloud information, (Meekins, paragraph 73 teaches "an Iterative Closest Point (ICP) process fine tunes the point translation and determines a rotation. Stated in another way, the platform 102 aligns the triangulated points and the target points about a common centroid. A cross product between the two sets of “matching” points results in a 3×3 matrix"); this shows matching the two (first and second) point cloud information, this is based on ICP which is preset point cloud matching algorithm, and all of this leads to the second fixed-point rotation matrix being acquired as mentioned above; wherein the point cloud matching algorithm includes: iterative closest points (ICP) algorithm (Meekins, paragraph 73 teaches "an Iterative Closest Point (ICP) process fine tunes the point translation and determines a rotation. Stated in another way, the platform 102 aligns the triangulated points and the target points about a common centroid. A cross product between the two sets of “matching” points results in a 3×3 matrix "); this shows ICP algorithm for point cloud matching. The same motivations used in claim 23 apply here in claim 24.
Regarding claim 25, the combination of Ojea, Zhang, Zeng, Zhu and Meekins teaches wherein the determining the rotation matrix based on the first fixed-point rotation matrix and the second fixed-point rotation matrix includes: serving the product of the first fixed-point rotation matrix and the second fixed-point rotation matrix as the third fixed-point rotation matrix corresponding to the fixed-point depth camera equipment (Meekins, paragraph 73 teaches "The U and V matrices can be multiplied, ignoring the S matrix, and a resulting rotation matrix can be used to move the point cloud towards the target points. This resulting matrix can be applied to all points to be rotated"); This shows matrices U and V which are first and second fixed-point rotation matrices from above, the resulting rotation matrix here (product) also acts as third fixed -point rotation matrix (fixed-point and corresponding to fixed-point depth camera equipment when viewed in combination); and calculating the rotation matrix by using the point cloud matching algorithm and taking the third fixed-point rotation matrix as an initial value, (Meekins, paragraph 73 teaches "an Iterative Closest Point (ICP) process...a resulting rotation matrix"); resulting rotation matrix here is achieved by using ICP (point cloud matching algorithm) and if it[rotation matrix] itself is the third fixed-point rotation matrix as described above, the initial value taken is used to simply refine/optimize as one of ordinary skill in the art would understand thus leading to a final resulting rotation matrix; wherein the point cloud matching algorithms include: ICP algorithm (Meekins, paragraph 73 teaches "an Iterative Closest Point (ICP) process fine tunes the point translation and determines a rotation. Stated in another way, the platform 102 aligns the triangulated points and the target points about a common centroid. A cross product between the two sets of “matching” points results in a 3×3 matrix "); this shows ICP algorithm for point cloud matching. The same motivations used in claim 23 apply here in claim 25.
Claim(s) 36-38 is/are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Ojea, Zhang, Zeng and Li as applied to claim 26 above, and further in view of Adam.
Regarding claim 36, the combination of Ojea, Zhang, Zeng and Li fails to teach wherein the generating the scene space model based on the first point cloud information and the second panoramic image includes: performing surface reconstruction processing on the first point cloud information based on a surface reconstruction algorithm to generate a mesh model corresponding to the first point cloud information; generating a texture of the mesh based on position information of the mesh of the mesh model and the second panoramic image; and setting the texture at corresponding mesh to generate a three-dimensional space model.
However, Adam teaches wherein the generating the scene space model based on the first point cloud information and the second panoramic image includes: performing surface reconstruction processing on the first point cloud information based on a surface reconstruction algorithm to generate a mesh model corresponding to the first point cloud information (Adam, paragraph 66 teaches "a mesh M is generated as illustrated by FIG. 1E from the point clouds obtained,...according to conventional methods, such as by reconstruction of Poisson surfaces "); reconstruction of Poisson surfaces shows surface reconstruction algorithm which generates the mesh that's from (corresponds to) the point cloud information (first point cloud information from Ojea when viewed in combination), thus the surface reconstruction processing of the first point cloud information is based on such; generating a texture of the mesh based on position information of the mesh of the mesh model and the second panoramic image (Adam, paragraph 66 teaches "a mesh M is generated as illustrated by FIG. 1E from the point clouds obtained, and a texture intended to be applied thereto "); since texture is applied to mesh, it must be based on position information and when viewed in combination this is also based on second panoramic image from Zeng since that image is constructed after rotation, whereas this is a reconstruction which would occur after initial construction; and setting the texture at corresponding mesh to generate a three-dimensional space model (Adam, claim 1 teaches "volumetric video stream of a three-dimensional action scene represented by a plurality of frames, the method being implemented by computer, and according to which a mesh and a texture for each frame of the scene are generated"); 3D action scene shows 3D space model being generated and having it be in accordance to mesh and texture shows texture would have to be set at the corresponding mesh. Adam is considered to be analogous art because it is reasonably pertinent to the problem faced by the inventor of surface reconstruction using mesh and texture. Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination of Ojea, Zhang, Zeng and Li with the reconstruction, mesh and texture techniques of Adam to improve firstly the compression of the information relating to the meshes of the frames of a volumetric video (Adam, paragraph 27). This would create a more efficient system and would be done due to the techniques of the specific surface reconstruction.
Regarding claim 37, the combination of Ojea, Zhang, Zeng, Meekins and Adam teaches wherein the generating the scene space model based on the first point cloud information and the second panoramic image includes: performing surface reconstruction processing on the first point cloud information based on a surface reconstruction algorithm to generate a mesh model corresponding to the first point cloud information (Adam, paragraph 66 teaches "a mesh M is generated as illustrated by FIG. 1E from the point clouds obtained,...according to conventional methods, such as by reconstruction of Poisson surfaces "); reconstruction of Poisson surfaces shows surface reconstruction algorithm which generates the mesh that's from (corresponds to) the point cloud information (first point cloud information from Ojea when viewed in combination), thus the surface reconstruction processing of the first point cloud information is based on such; generating a texture of the mesh based on position information of the mesh of the mesh model and the second panoramic image (Adam, paragraph 66 teaches "a mesh M is generated as illustrated by FIG. 1E from the point clouds obtained, and a texture intended to be applied thereto "); since texture is applied to mesh, it must be based on position information and when viewed in combination this is also based on second panoramic image from Zeng since that image is constructed after rotation, whereas this is a reconstruction which would occur after initial construction; and setting the texture at corresponding mesh to generate a three-dimensional space model (Adam, claim 1 teaches "volumetric video stream of a three-dimensional action scene represented by a plurality of frames, the method being implemented by computer, and according to which a mesh and a texture for each frame of the scene are generated"); 3D action scene shows 3D space model being generated and having it be in accordance to mesh and texture shows texture would have to be set at the corresponding mesh. The same motivations used in claim 36 apply here in claim 37.
Regarding claim 38, the combination of Ojea, Zhang, Zeng, Meekins and Adam teaches wherein the generating the scene space model based on the first point cloud information and the second panoramic image includes: performing surface reconstruction processing on the first point cloud information based on a surface reconstruction algorithm to generate a mesh model corresponding to the first point cloud information (Adam, paragraph 66 teaches "a mesh M is generated as illustrated by FIG. 1E from the point clouds obtained,...according to conventional methods, such as by reconstruction of Poisson surfaces "); reconstruction of Poisson surfaces shows surface reconstruction algorithm which generates the mesh that's from (corresponds to) the point cloud information (first point cloud information from Ojea when viewed in combination), thus the surface reconstruction processing of the first point cloud information is based on such; generating a texture of the mesh based on position information of the mesh of the mesh model and the second panoramic image (Adam, paragraph 66 teaches "a mesh M is generated as illustrated by FIG. 1E from the point clouds obtained, and a texture intended to be applied thereto "); since texture is applied to mesh, it must be based on position information and when viewed in combination this is also based on second panoramic image from Zeng since that image is constructed after rotation, whereas this is a reconstruction which would occur after initial construction; and setting the texture at corresponding mesh to generate a three-dimensional space model (Adam, claim 1 teaches "volumetric video stream of a three-dimensional action scene represented by a plurality of frames, the method being implemented by computer, and according to which a mesh and a texture for each frame of the scene are generated"); 3D action scene shows 3D space model being generated and having it be in accordance to mesh and texture shows texture would have to be set at the corresponding mesh. The same motivations used in claim 36 apply here in claim 38.
Claim(s) 31-35 is/are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Ojea, Zhang, Zeng, Zhu and Meekins as applied to claim 23-25 above, and further in view of Adam.
Regarding claim 31, the combination of Ojea, Zhang, Zeng and Meekins fails to teach wherein the generating the scene space model based on the first point cloud information and the second panoramic image includes: performing surface reconstruction processing on the first point cloud information based on a surface reconstruction algorithm to generate a mesh model corresponding to the first point cloud information; generating a texture of the mesh based on position information of the mesh of the mesh model and the second panoramic image; and setting the texture at corresponding mesh to generate a three-dimensional space model.
However, Adam teaches wherein the generating the scene space model based on the first point cloud information and the second panoramic image includes: performing surface reconstruction processing on the first point cloud information based on a surface reconstruction algorithm to generate a mesh model corresponding to the first point cloud information (Adam, paragraph 66 teaches "a mesh M is generated as illustrated by FIG. 1E from the point clouds obtained,...according to conventional methods, such as by reconstruction of Poisson surfaces "); reconstruction of Poisson surfaces shows surface reconstruction algorithm which generates the mesh that's from (corresponds to) the point cloud information (first point cloud information from Ojea when viewed in combination), thus the surface reconstruction processing of the first point cloud information is based on such; generating a texture of the mesh based on position information of the mesh of the mesh model and the second panoramic image (Adam, paragraph 66 teaches "a mesh M is generated as illustrated by FIG. 1E from the point clouds obtained, and a texture intended to be applied thereto "); since texture is applied to mesh, it must be based on position information and when viewed in combination this is also based on second panoramic image from Zeng since that image is constructed after rotation, whereas this is a reconstruction which would occur after initial construction; and setting the texture at corresponding mesh to generate a three-dimensional space model (Adam, claim 1 teaches "volumetric video stream of a three-dimensional action scene represented by a plurality of frames, the method being implemented by computer, and according to which a mesh and a texture for each frame of the scene are generated"); 3D action scene shows 3D space model being generated and having it be in accordance to mesh and texture shows texture would have to be set at the corresponding mesh. Adam is considered to be analogous art because it is reasonably pertinent to the problem faced by the inventor of surface reconstruction using mesh and texture. Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination of Ojea, Zhang, Zeng and Meekins with the reconstruction, mesh and texture techniques of Adam to improve firstly the compression of the information relating to the meshes of the frames of a volumetric video (Adam, paragraph 27). This would create a more efficient system and would be done due to the techniques of the specific surface reconstruction.
Regarding claim 32, the combination of Ojea, Zhang, Zeng, Meekins and Adam teaches wherein the surface reconstruction algorithm includes: Possion surface reconstruction algorithm (Adam, paragraph 66 teaches "such as by reconstruction of Poisson surfaces"); this shows Possion surface reconstruction algorithm; the mesh includes: a triangular mesh and a quadrilateral mesh (Adam, paragraph 7 teaches "as a basis for the modeling of the scene by a mesh of triangles connected continuously to one another, onto which a texture is applied" and paragraph 82 teaches "Block compression is a conventional compression method making it possible to reduce the quantity of memory required to store color data, wherein blocks of pixels (such as squares of 4 pixels by 4 pixels or 8 pixels by 8 pixels) are compressed"); this shows triangular mesh and quadrilateral mesh included in the mesh since one of ordinary skill in the art would understand that the block compression technique used for the texture to get square/quadrilateral blocks would also be used for the mesh in cases and places where the squares better fit the mesh in lieu of the triangle and lead to a more smoothened output. The same motivations used in claim 31 apply here in claim 32.
Regarding claim 33, the combination of Ojea, Zhang, Zeng, Meekins and Adam teaches wherein the generating the scene space model based on the first point cloud information and the second panoramic image includes: performing surface reconstruction processing on the first point cloud information based on a surface reconstruction algorithm to generate a mesh model corresponding to the first point cloud information (Adam, paragraph 66 teaches "a mesh M is generated as illustrated by FIG. 1E from the point clouds obtained,...according to conventional methods, such as by reconstruction of Poisson surfaces "); reconstruction of Poisson surfaces shows surface reconstruction algorithm which generates the mesh that's from (corresponds to) the point cloud information (first point cloud information from Ojea when viewed in combination), thus the surface reconstruction processing of the first point cloud information is based on such; generating a texture of the mesh based on position information of the mesh of the mesh model and the second panoramic image (Adam, paragraph 66 teaches "a mesh M is generated as illustrated by FIG. 1E from the point clouds obtained, and a texture intended to be applied thereto "); since texture is applied to mesh, it must be based on position information and when viewed in combination this is also based on second panoramic image from Zeng since that image is constructed after rotation, whereas this is a reconstruction which would occur after initial construction; and setting the texture at corresponding mesh to generate a three-dimensional space model (Adam, claim 1 teaches "volumetric video stream of a three-dimensional action scene represented by a plurality of frames, the method being implemented by computer, and according to which a mesh and a texture for each frame of the scene are generated"); 3D action scene shows 3D space model being generated and having it be in accordance to mesh and texture shows texture would have to be set at the corresponding mesh. The same motivations used in claim 31 apply here in claim 33.
Regarding claim 34, the combination of Ojea, Zhang, Zeng, Meekins and Adam teaches wherein the surface reconstruction algorithm includes: Possion surface reconstruction algorithm (Adam, paragraph 66 teaches "such as by reconstruction of Poisson surfaces"); this shows Possion surface reconstruction algorithm; the mesh includes: a triangular mesh and a quadrilateral mesh (Adam, paragraph 7 teaches "as a basis for the modeling of the scene by a mesh of triangles connected continuously to one another, onto which a texture is applied" and paragraph 82 teaches "Block compression is a conventional compression method making it possible to reduce the quantity of memory required to store color data, wherein blocks of pixels (such as squares of 4 pixels by 4 pixels or 8 pixels by 8 pixels) are compressed"); this shows triangular mesh and quadrilateral mesh included in the mesh since one of ordinary skill in the art would understand that the block compression technique used for the texture to get square/quadrilateral blocks would also be used for the mesh in cases and places where the squares better fit the mesh in lieu of the triangle and lead to a more smoothened output. The same motivations used in claim 31 apply here in claim 34.
Regarding claim 35, the combination of Ojea, Zhang, Zeng, Meekins and Adam teaches wherein the generating the scene space model based on the first point cloud information and the second panoramic image includes: performing surface reconstruction processing on the first point cloud information based on a surface reconstruction algorithm to generate a mesh model corresponding to the first point cloud information (Adam, paragraph 66 teaches "a mesh M is generated as illustrated by FIG. 1E from the point clouds obtained,...according to conventional methods, such as by reconstruction of Poisson surfaces "); reconstruction of Poisson surfaces shows surface reconstruction algorithm which generates the mesh that's from (corresponds to) the point cloud information (first point cloud information from Ojea when viewed in combination), thus the surface reconstruction processing of the first point cloud information is based on such; generating a texture of the mesh based on position information of the mesh of the mesh model and the second panoramic image (Adam, paragraph 66 teaches "a mesh M is generated as illustrated by FIG. 1E from the point clouds obtained, and a texture intended to be applied thereto "); since texture is applied to mesh, it must be based on position information and when viewed in combination this is also based on second panoramic image from Zeng since that image is constructed after rotation, whereas this is a reconstruction which would occur after initial construction; and setting the texture at corresponding mesh to generate a three-dimensional space model (Adam, claim 1 teaches "volumetric video stream of a three-dimensional action scene represented by a plurality of frames, the method being implemented by computer, and according to which a mesh and a texture for each frame of the scene are generated"); 3D action scene shows 3D space model being generated and having it be in accordance to mesh and texture shows texture would have to be set at the corresponding mesh. The same motivations used in claim 31 apply here in claim 35.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Huang et al. (U.S. Patent Application Publication No. 2021/0223048) paragraph 27 teaches “the pose of the point cloud acquisition device refers to absolute coordinates of the point cloud acquisition device (e.g., GPS (Global Positioning System) coordinates) and a posture of the point cloud acquisition device (e.g., camera external parameters, including a rotation matrix and a translation matrix) when the point cloud is acquired”; this shows rotation matrix of point cloud acquisition device (camera)'s absolute coordinates (camera coordinate system) and corresponds to GPS thus global coordinate (and is when point cloud is acquired thus based on both point clouds).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NAUMAN U AHMAD whose telephone number is (703)756-5306. The examiner can normally be reached Monday - Friday 9:00am - 5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee Tung can be reached at (571) 272-7794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/N.U.A./Examiner, Art Unit 2611
/KEE M TUNG/Supervisory Patent Examiner, Art Unit 2611