DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on/after Mar. 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Allowable Subject Matter
Claims 6-7 and 20-21 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including ALL of the limitations of the base claim AND any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter:
Regarding claim 6 and claim 20, the prior art of record does not teach, suggest, or disclose the claim limitation “determining, in a case that a quantity of other point cloud data with distances to the straight-line data less than a fifth threshold satisfies a quantity condition, the straight-line data as the boundary line data between the any two first point cloud clusters” in combination with the recited limitations inherited from the base claim and the intervening claims, as well as the other limitations recited amongst the cited limitation.
Regarding claim 7 and claim 21, the prior art of record does not teach, suggest, or disclose the claim limitation “determining, in a case that each second point cloud cluster satisfies the partitioning condition, boundary line data between every two first point cloud clusters and boundary line data between every two second point cloud clusters as at least one piece of boundary line data” in combination with the recited limitations inherited from the base claim and the intervening claims, as well as the other limitations recited amongst the cited limitation.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1, 9, and 11 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Xiao [JIANXIONG] (U.S. PG-PUB 2022/0318456, 'XIAO-2022').
Regarding claim 1, XIAO-2022 discloses a method for rendering an image, comprising:
acquiring … pieces of raw point cloud data by scanning an object with a scanning device (XIAO-2022; FIG. 1; ¶ 0022-24; “The point cloud data is acquired by radar devices, such as lidar … The point cloud data … maps 3D environment, and detects objects.”), wherein any one piece of raw point cloud data … describes a data acquisition point on the object (XIAO-2022; FIG. 1, FIG. 5; ¶ 0026; “In step S103, … [3-D] contours are … rendered according to the … [2-D] contours and the point cloud data. Each [3-D] contour presents a [3-D] object, and each [3-D] contour is formed by adding spatial data to the corresponding [2-D] contour, and has a size and a color [‘data acquisition point on the object’] …”); [and]
determining … piece(s) of boundary line data based on the … pieces of raw point cloud data (XIAO-2022; FIGS. 3-4; ¶ 0037-38; “In the step S10232, … boundary contour regions of … first grayscale images are calculated according to a preset boundary algorithm. The boundary contour area is a part of the boundary of the object. … a binarization of the basis of the pre-processed first grayscale image is performed based on a hysteresis threshold [which] can ensure that the final contour image is continuous by using a recursive tracking algorithm. The binary process is regularly associated with standard contours, such as artificially drawn contours in an image database. In … other embodiments, another general method for obtaining the boundary contour area includes segmenting the images, and then the boundary of the segmented area is directly used as the boundary contour area. In the step S10233, … the … contours are calculated from the … boundary contour regions according to the principle of maximum area.”), wherein any one piece of boundary line data … describes a contour boundary on the object (XIAO-2022; FIGS. 5-6; ¶ 0040-44; “… a flowchart of sub-steps of step S103 is illustrated … How to render … [3-D] contours according to … [2-D] contours and point cloud data will be described … In the step S1031, a second label to each of the … objects is added according to the point cloud data to obtain … second labels. … In the step S1032, the … first labels and … the second labels are matched to generate … pairs of [3-D] information. … a first label indicating a vehicle and a second label indicating a vehicle are matched as a pair of [3-D] information. The … first labels in the image data and the … second labels in the point cloud data are matched into the … [3-D] information pairs. [0043] In the step S1033, … [3-D] contours are rendered according to the information contained in the … [3-D] information pairs. … the [3-D] information in the [3-D] data is supplemented into the [2-D] contour. … the [3-D] contours are capable of providing the autonomous vehicle with more spatial information of obstacles. Comparing with the existing [2-D] contours, the [3-D] contours can provide more effective information. … the situation of a traffic light being blocked by a large truck in front, the situation of [a person] suddenly running out from the front of a large truck parked on the side of the road, cannot be solved by the data provided by the [2-D] contour. The [3-D] contour can provide spatial data about the truck …, so that the [AV] system can calculate the spatial relationship between the traffic lights and the obstacle truck and the [AV] itself, and carry out the next trajectory planning according to the spatial relationship. … although [2-D] contours can also detect ghost probes, the lack of distance dimension information makes the [AV] unable to make correct decisions. The [3-D] contour can accurately provide distance information, and the [AV] can make correct decisions.”); and
determining an object image based on the … pieces of raw point cloud data and the ... piece(s) of boundary line data (XIAO-2022; ¶ 0026; “In step S103, … [3-D] contours are … rendered according to the … [2-D] contours and the point cloud data. Each [3-D] contour presents a [3-D] object, and each [3-D] contour is formed by adding spatial data to the corresponding [2-D] contour, and has a size and a color … How to perform the step of acquiring the [3-D] contour will be described in the following steps of S1031-S1033.” FIG. 5; ¶ 0040-43; “… a flowchart of sub-steps of step S103 is illustrated … How to render … [3-D] contours according to … [2-D] contours and point cloud data will be described in the following steps of S1031-S1033. … In the step S1031, a second label to each of the … objects are added according to the point cloud data to obtain … second labels. How to acquire … second labels will be described of steps S10311-S10312. In the step S1032, the … first labels and the … second labels are matched to generate … pairs of [3-D] information. In detail, a first label indicating a vehicle and a second label indicating a vehicle are matched as a pair of [3-D] information. The … first labels in the image data and the … second labels in the point cloud data are matched into the … [3-D] information pairs. In the step S1033, … [3-D] contours are rendered according to the information contained in the … [3-D] information pairs. In detail, the [3-D] information in the [3-D] data is supplemented into the [2-D] contour.”), and
rendering the object image (XIAO-2022; FIG. 1; ¶ 0027; “In step S104, simulation is performed according to the … [3-D] contours … [which] are shapes of the [3-D] object but not appearances of the objects. Simulation software recognizes obstacles to what objects or types of the objects via [3-D] contours, so as to achieve near-perfect graphics perception [‘rendering the object image’] and radar perception.”).
Regarding claim 9, XIAO-2022 discloses an apparatus for rendering an image, comprising:
a processor; and a memory … stores … instruction(s) executable by the processor (XIAO-2022; FIG. 7, ‘Computer Device 900’, ‘Memory 901’, and ‘Processor 902’; ¶ 0055-58); wherein
the processor, when loading and executing the … instruction(s) (XIAO-2022; ¶ 0007; “… the disclosure provides a computer equipment [which] comprises: a memory configured to store program instructions and a processor configured to execute the program instructions to enable the computer equipment to perform the simulation method based on [3-D] contours …”), is caused to perform: … ([The remaining limitations are repeated nearly verbatim from those recited in independent claim 1; please see their treatment in the rejection of claim 1 in the Office action above.]).
Regarding claim 11, XIAO-2022 discloses a non-transitory computer-readable storage medium storing … instruction(s) therein, wherein the … instruction(s), when loaded and executed by a processor of an electronic device for rendering an image (XIAO-2022; ¶ 0064; “… the simulation method [‘rendering an image’] based on [3-D] contours includes … program instruction(s). When the program instructions are loaded and executed on the computer device 900, the procedures or functions of the embodiments of the present invention are generated in whole or in part. … The computer-readable storage medium can be any available medium that can be stored by a computer, or a data storage device such as a server, data center, etc., which includes … available media integrated. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVD), or semiconductor media (e.g., … (SSD)), [etc.] [‘non-transitory computer-readable storage media’]”), cause the electronic device to perform: … ([The remaining limitations are repeated verbatim from those recited in independent claim 1; please see their treatment in the rejection of claim 1 in the Office action above.]).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 USC 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 2, 4, 14, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over XIAO-2022 as applied to claims 1 and 9 above, respectively, and further in view of Siao et al. (U.S. Patent 10,634,793; 'SIAO').
Regarding claim 2 and claim 14, XIAO-2022 discloses the method for rendering the image according to claim 1 and the apparatus for rendering the image according to claim 9; however, XIAO-2022 does not explicitly disclose that said determining … piece(s) of boundary line data based on the … pieces of raw point cloud data comprise:
acquiring … pieces of target point cloud data by filtering out raw point cloud data satisfying a filtering condition from the … pieces of raw point cloud data, which SIAO discloses (SIAO; FIG. 4; Col. 6, Lines 15-25; “The operation of the lidar detection device 10 … is described … In a step S10, the four [2-D] lidars 14 are used to scan all obstacles, to obtain original point-cloud data OD corresponding to the all obstacles, and the original point-cloud data OD includes the relative distance, the relative angle and the relative speed of each obstacle relative to the vehicle 12. Next, in a step S12, the noise filter 16 receives the original point-cloud data OD and filters out the noise of the original point-cloud data OD [‘filtering out raw point cloud data’], to generate the filtered point-cloud data FD [‘acquiring … pieces of target point cloud data’].”); and
determining … piece(s) of boundary line data based on the … pieces of target point cloud data, which SIAO also discloses (SIAO; FIG. 4; Col. 6, Lines 25-31; “In a step S14, the processor 18 receives the filtered point-cloud data FD and classifies, by the preset length, the filtered point-cloud data FD [‘pieces of target point cloud data’] into … point-cloud groups corresponding to … all obstacles, … and obtains the border length of each obstacle according to the contour of each point-cloud group corresponding to the obstacle.”).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to modify the method for rendering the image according to claim 1 and the apparatus for rendering the image according to claim 9 of XIAO-2022 to include the acquiring … pieces of target point cloud data by filtering out raw point cloud data satisfying a filtering condition from the … pieces of raw point cloud data and the determining … piece(s) of boundary line data based on the … pieces of target point cloud data of SIAO. The motivation for this modification is to classify point-cloud data into point-cloud group(s) corresponding to obstacle(s), and border length(s) of the obstacle(s) is/are obtained according to contour(s) of the point-cloud group. Kalman filtering and extrapolation are used to estimate and track movement path(s) of dynamic obstacle(s), and transmit the border length(s) of the dynamic obstacle(s). According to the relative distance(s), a coordinate of the dynamic obstacle(s) nearest the vehicle/viewer can be obtained and transmitted efficiently (SIAO; Abstract).
XIAO-2022*SIAO disclose that said determining the object image based on the … pieces of raw point cloud data and the … piece(s) of boundary line data comprises:
determining the object image based on the … pieces of target point cloud data and the … piece(s) of boundary line data (XIAO-2022; FIGS. 1, 5; ¶ 0026-27; “In step S103, … [3-D] contours are according to rendered according to the … [2-D] contours [‘boundary line data’] and the point cloud data. Each [3-D] contour presents a [3-D] object, and each [3-D] contour is formed by adding spatial data to the corresponding [2-D] contour, and has a size and a color without other information. How to perform the step of acquiring the [3-D] contour will be described in the following steps of S1031-S1033. In step S104, simulation is performed according to the … [3-D] contours … [which] are shapes of the [3-D] object but not appearances of the objects. Simulation software recognizes obstacles to what objects or types of the objects via [3-D] contours, so as to achieve near-perfect graphics perception [‘determining the object image’] and radar perception.”).
Regarding claim 4 and claim 18, XIAO-2022*SIAO disclose the method for rendering the image according to claim 2 and the apparatus for rendering the image according to claim 14, wherein said determining … piece(s) of boundary line data based on the … pieces of target point cloud data comprise:
acquiring at least two first point cloud clusters by partitioning the … pieces of target point cloud data, wherein any first point cloud cluster comprises … piece(s) of target point cloud data;
determining, for any two first point cloud clusters, boundary line data between the any two first point cloud clusters based on each piece of target point cloud data in the any two first point cloud clusters (SIAO; FIG. 4; Col. 6, Lines 15-31; “The operation of the lidar detection device 10 … is described … In a step S10, the four [2-D] lidars 14 are used to scan all obstacles, to obtain original point-cloud data OD corresponding to the all obstacles, and the original point-cloud data OD includes the relative distance, the relative angle and the relative speed of each obstacle relative to the vehicle 12 [‘acquiring … point cloud clusters’]. Next, in a step S12, the noise filter 16 receives the original point-cloud data OD and filters out the noise of the original point-cloud data OD, to generate the filtered point-cloud data FD [‘target point cloud data’]. In a step S14, the processor 18 receives the filtered point-cloud data FD and classifies, by the preset length, the filtered point-cloud data FD into … point-cloud groups [‘partitioning the … pieces of target point cloud data’] corresponding to the all obstacles, respectively, and obtains the border length of each obstacle according to the contour of each point-cloud group corresponding to the obstacle [‘determining … boundary line data between the any two first point cloud clusters’].”); and
determining, in a case that the at least two first point cloud clusters satisfy a partitioning condition, boundary line data between every two first point cloud clusters as … piece(s) of boundary line data (XIAO-2022; FIG. 4; ¶ 0037; “In the step S10232, … boundary contour regions of … first grayscale images are calculated according to a preset boundary algorithm. The boundary contour area is a part of the boundary of the object. … a binarization of the basis of the pre-processed first grayscale image is performed based on a hysteresis threshold [which] can ensure that the final contour image is continuous by using a recursive tracking algorithm. The binary process is regularly associated with standard contours, such as artificially drawn contours in an image database. In … other embodiments, another general method for obtaining the boundary contour area includes segmenting the images, and then the boundary of the segmented area is directly used as the boundary contour area. [0038] In the step S10233, … contours are calculated according to the … boundary contour regions. … the … contours are calculated from the … boundary contour regions according to the principle of maximum area.”).
Claims 3 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over XIAO-2022 in view of SIAO as applied to claims 2 and 14 above, respectively, and further in view of Xiao [YONG] et al. (U.S. PG-PUB 2021/0334988, 'XIAO-2021').
Regarding claim 3 and claim 15, XIAO-2022*SIAO disclose the method for rendering the image according to claim 2 and the apparatus for rendering the image according to claim 14; however, XIAO-2022*SIAO do not explicitly disclose the following limitations, which XIAO-2021 discloses:
PNG
media_image1.png
470
562
media_image1.png
Greyscale
the any one piece of raw point cloud data comprises coordinate data (XIAO-2021; ¶ 0048; ¶ 0080; “Point clouds receiver module 401 can receive point clouds (e.g., LIDAR images captured by LIDAR sensors of an ADV) and corresponding poses (e.g., position and orientation). A point cloud refers to a set of data points … captured by a LIDAR device at a particular point in time. Each data point is associated with location information of the data point (e.g., xyz coordinates). Point clouds down-sampling module 403 … can down-sample the point clouds spatially [‘filtering condition comprises raw point cloud data with coordinate data outside of a set area’] or temporally. … Partition module 407 can partition … navigable area(s) into … partition(s) based on the closure information or block partition information. Optimization solver module 409 can apply an optimization algorithm (such as a bundle adjustment or an ADMM algorithm, as part of algorithms 415 of FIG. 4 or algorithms 124 of FIG. 1) to point clouds and poses corresponding to a partition to generate refined HD LIDAR poses.”); and
the raw point cloud data satisfying the filtering condition comprises raw point cloud data with coordinate data outside of a set area (XIAO-2021; FIG. 6; ¶ 0084; “… process 601 can down-sample point clouds spatially. … process 601 can determine a spatial sliding window [which] can be [a] circular window with a predetermined radius. Process 601 then identifies one LIDAR point cloud and a corresponding pose for all point clouds inside the spatial window for further processing. [Otherwise], process 601 can dispose of the other LIDAR point clouds and their corresponding poses which are not identified for further processing. … the one LIDAR point cloud [is] identified based on a spatial region of interest, e.g., a central region of the sliding window.” FIG. 24; ¶ 151; “… multi-scale directional difference features [are] extracted by: 1) For each point (e.g., P), setting a small neighborhood radius r, a large neighborhood radius R, and identifying points sets Nsmall or Ns and Nlarge or Nl respectively within the corresponding radius neighborhood. … the points in Nl [are] inclusive of the points in Ns.” [The Examiner notes that the aforementioned spatial down-sampling of point cloud(s) may be accomplished using the neighborhoods depicted in FIG. 24.]).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to modify the method for rendering the image according to claim 2 and the apparatus for rendering the image according to claim 14 of XIAO-2022*SIAO to include the disclosure that any one piece of raw point cloud data comprises coordinate data and the disclosure that the raw point cloud data satisfying the filtering condition comprises raw point cloud data with coordinate data outside of a set area of XIAO-2021. The motivation for this modification is to partition large sets of 3-D point cloud data by determining central tendencies to cluster points in a concentrated area, and to exclude/partition points exterior to the first cluster of points into a separate cluster of points. This partitioning improves the coherency of distinct clusters within point clouds and structures large point cloud data sets.
Claims 5 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over XIAO-2022 in view of and SIAO as applied to claims 4 and 18 above, respectively, and further in view of Najaf-Zadeh et al. (U.S. PG-PUB 2020/0204782, 'NAJAF-ZADEH').
Regarding claim 5 and claim 19, XIAO-2022*SIAO disclose the method for rendering the image according to claim 4 and the apparatus for rendering the image according to claim 18; however, XIAO-2022*SIAO do not explicitly disclose that said acquiring … first point cloud clusters by partitioning the … pieces of target point cloud data comprise the following limitations, which NAJAF-ZADEH discloses:
determining, for any two pieces of target point cloud data, a distance between the any two pieces of target point cloud data based on coordinate data in the any two pieces of target point cloud data (NAJAF-ZADEH; FIG. 7B; ¶ 0029-30, 0035, 0070, 0077, 0086, 0109, 0145, 0163; “The 3D cell 720 … includes a query point 730. The 3D cell 720 can be included in the grid 700. The 3D cell 720 can include multiple points of a reconstructed 3D point cloud, but only a single query point is illustrated. Prior to reconstructing the point cloud, the query point 730 was represented as a pixel that was at a border of one of the patches within a frame. The query point 730 is located at coordinates (XYZ).”; FIGS. 5C; ¶ 0114; “The smoothing engine 566 also excludes neighboring cells whose color is too different than the cell with the query point. … the smoothing engine 566 identifies a distance between the luminance value (or color centroid) of each neighboring cell to the cell with the query point and compares the distance to a threshold. … if the difference between the luminance value (or color centroid) of the cell containing the query point and a luminance value (or color centroid) of a neighboring cell is greater than a threshold, that neighboring cell is excluded from the group of selected neighboring cells.”); and
partitioning, in a case that a difference between color data in the any two pieces of target point cloud data is less than a first threshold and a distance between the any two pieces of target point cloud data is less than a second threshold, the any two pieces of target point cloud data into a same first point cloud cluster (NAJAF-ZADEH; FIGS. 5C, 6; ¶ 106-107; “The smoothing engine 566 generates a 3D grid and places the reconstructed point cloud within the grid [which] is composed of multiple non-overlapping cells. The shape of each cell can be the same for every cell within the grid or the shape can vary from cell to cell. … the size of each cell within the grid can be the same. The size of the cells affects the level of smoothing. … larger cells can cause more smoothing to occur.… the size of each cell varies. … the cells can be positioned over a portion of the point cloud with similar colors. That is, [the] point cloud can be split into a non-uniform grid. Grouping points in a non-uniform grid would cluster points with similar color in geometric proximity to each other into a single cell. Grouping points of like colors into a single cell avoids the possibility of mixing up points with very different color into the same cell. … smoothing engine 566 can generate a uniform grid with large cells. After generating a grid with large cells, … smoothing engine 566 splits … cells into smaller cells depending on the color variations within each cell to avoid large color variation inside cells.” ¶ 118; “After generating the new color, … smoothing engine 566 compares the color of the query point to the new color. If the distance between the color of the query point and the new color is less than a threshold, no smoothing is performed. … smoothing engine 566 does not modify the color of the query point since the color of the query point is determined to be similar to the new color … If the distance between the color of the query point and the new color is greater than a threshold, the smoothing engine 566 replaces the color of the query point with the new color.”).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to modify the method for rendering the image according to claim 4 and the apparatus for rendering the image according to claim 18 of XIAO-2022*SIAO to include the determining, for any two pieces of target point cloud data, a distance between the any two pieces of target point cloud data based on coordinate data in the any two pieces of target point cloud data and the partitioning, in a case that a difference between color data in the any two pieces of target point cloud data is less than a first threshold and a distance between the any two pieces of target point cloud data is less than a second threshold, the any two pieces of target point cloud data into a same first point cloud cluster of NAJAF-ZADEH. The motivation for this modification is to perform point cloud compression and decompression using a video codec. When a 3-D point cloud is converted from a 3-D representation to a 2-D representation, the points of 3D point cloud are clustered into groups and projected onto frames, where the clustered points result in patches that are packed onto 2-D frames. Due to the size constraints of certain 2-D frames, two patches that are not next to each other on the 3-D point cloud can be packed next to each other in a single frame. When two non-neighboring patches of the point cloud are packed next to each other in a 2-D frame, the pixels from one patch can be inadvertently mixed up with the pixels from the other patch by the block-based video codec. When pixels from one patch are inadvertently included in another patch, visible artifacts can occur at patch boundaries when the point cloud is reconstructed by the decoder. Therefore, smoothing the color of the points near the patch boundary avoids visual artifacts. Removing visual artifacts improves the visual quality of the point cloud. Smoothing the color component of the point cloud at the decoder can create similar visual quality at a lower bit-rate (NAJAF-ZADEH; ¶ [0037]). Lowering the bit-rate of the point cloud data allows for faster transmissions and more efficient storage, potentially yielding real-time applications on mobile devices.
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over XIAO-2022 as applied to claim 1 above, and further in view of Sandrew et al. (U.S. PG-PUB 2015/0358613, 'SANDREW').
Regarding claim 8, XIAO-2022 discloses the method for rendering the image according to claim 1; however, XIAO-2022 does not explicitly disclose that said rendering the object image comprises the following features, which SANDREW discloses/teaches:
PNG
media_image2.png
510
676
media_image2.png
Greyscale
acquiring an offset between a reference point of a display screen on a head-mounted display device and a reference point of the object image (SANDREW; FIG. 20; ¶ 0106; “The [UI] provide … control(s) to edit the pixel translations of the selected objects. The illustrative [UI] 2001 … provides … an offset control 2005 that adds/subtracts a fixed number of pixels to the translations for the selected objects [‘acquiring an offset’] … The Apply Changes button 2008 applies the pixel translation modifications to the selected objects and regenerates the left and right images accordingly. In FIG. 20, the offset control 2005 is selected and the offset value 2010 is set to 10 pixels. FIG. 21 shows the effect of this modification. The images in FIG. 21 are anaglyph images with left and right images overlaid using different color filters. The top image 2101 shows a panoramic image prior to the modifications from FIG. 20. The bottom image 2103 shows the panoramic image after applying pixel translation modifications 2102. The couch, rocking chair, and table [‘reference point of the object image’] appear to be pulled further forward in the scene because the left and right pixel shifts [‘acquiring an offset’] for these objects have been increased.” ¶ 0117; “FIG. 26 … uses central dots 2611c, 2611d, and 2611e instead of boundaries around the entire fields of view. Each dot corresponds to the central region of the corresponding reviewer's field of view. Dots are color-coded as before, and in addition in this example are partially transparent so that the coordinator can observe objects through the dots. … One or more embodiments may combine the graphics of FIG. 25-26, to show … the entire field of view with a boundary and the center of the field of view with a dot [‘reference point of a display screen’] …”);
acquiring an offset object image by offsetting the object image based on the offset (SANDREW; FIGS. 18-19; ¶ 0100; “… the depth information 112 [‘offset’] is used … for updates to the stereoscopic images 115-116. Steps 1801-1802 are used for the initial 3D model … Depth information 112 is used to generate a spherical translation map 1901 [which] encodes information that determines the degree of horizontal pixel shifting [‘offset’] between the left/right eye images 115-116. … A spherical translation map may be … a parallax map, a disparity map, a depth map …”); and
rendering the offset object image on the display screen (SANDREW; FIG. 18; ¶ 0099; “The process of creating a 3D virtual environment may be iterative. … step 111a (Assign Depth Map to Points of Regions) may comprise two sub-steps: 1801 (Create 3D Model), and 1802 (Render). … creating the 3D model may … involve mapping images or regions to planes or other surfaces and positioning these planes or surfaces in 3D space. Rendering may … involve generating spherical or cylindrical projections from these planes and surfaces. After rendering, stereoscopic images 115-116 are generated in step 114 … Mapping of images onto 3D surfaces may … introduce artifacts that are not apparent until the stereoscopic images 115-116 are reviewed. In other cases, the images may be free of artifacts, but the depth assigned to regions or objects may not match the desired structure or artistic effects for the [VR] environment. … the review 1803 may in many cases lead to additional modifications 1801 to the 3D model. Normally these changes require an additional rendering step 1802, followed by another stereoscopic image generation step 114, and another review 1803.” FIG. 24; ¶ 0110; “Review stations may generate … display image(s) showing the 3D model from the viewpoint determined by the reviewer pose subsystem. Generating these images is the function of a viewpoint renderer. … viewpoint renderers may generate … stereoscopic views of the 3D model from the viewpoint corresponding to the reviewer's pose. … Display devices may … include computer monitors, as well as virtual reality headsets or other head-mounted displays. … viewpoint renderer 2413a in review station 2410a generates viewpoint images for display 2411a; viewpoint renderer 2413b in review station 2410b generates viewpoint images for display 2411b.”).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to modify the method for rendering the image according to claim 1 of XIAO-2022 to include the acquiring an offset between a reference point of a display screen on a head-mounted display device and a reference point of the object image, the acquiring an offset object image by offsetting the object image based on the offset, and the rendering the offset object image on the display screen of SANDREW. The motivation for this modification is to coordinate multiple simultaneous reviews of a 3-D model, potentially from different viewpoints, by supporting multiple reviewers using review stations that render images of the model based on the pose of the reviewer. Multiple reviewers may use VR headsets to observe a 3-D virtual environment from different orientations. A coordinator uses a coordinator station to observe the entire 3-D model and the viewpoints of each of the reviewers in this 3-D model. Real-time updates to the 3-D model and propagation of updated images to the coordinator and to the multiple viewers are supported (SANDREW; Abstract).
Claims 12-13 and 16-17 are rejected under 35 U.S.C. 103 as being unpatentable over XIAO-2022 in view of SIAO as applied to claims 2 and 14 above, respectively, and further in view of Mammou et al. (U.S. PG-PUB 2019/0087979, 'MAMMOU').
Regarding claim 12 and claim 16, XIAO-2022*SIAO disclose the method for rendering the image according to claim 2 and claim 14; however, XIAO-2022*SIAO do not explicitly disclose the following limitations, which MAMMOU discloses:
the any one piece of raw point cloud data comprises color data (MAMMOU; ¶ 0076-77; “… a system, may include … LIDAR system(s), 3-D camera(s)/scanner(s), etc., and such sensor devices may capture spatial information, such as XYZ coordinates for points in a view of the sensor devices. … the spatial information may be relative to a local coordinate system or may be relative to a global coordinate system (for example, a Cartesian coordinate system may have a fixed reference point, such as a fixed point on the earth, or may have a non-fixed local reference point, such as a sensor location). … such sensors may also capture attribute information for … point(s), such as color attributes, reflectivity attributes, velocity attributes, acceleration attributes, time attributes, modalities, and/or various other attributes. … other sensors, in addition to LIDAR systems, 3-D cameras/scanners, etc., may capture attribute information to be included in a point cloud.”); and
the raw point cloud data satisfying the filtering condition comprises raw point cloud data with color data matching a set color (MAMMOU; FIG. 4B, 4C; ¶ 0266; “… a closed-loop color conversion module, such as closed-loop color conversion module 410, receives a compressed point cloud from a video encoder, such as video compression module 218 illustrated in FIG. 4A or video compression module 264 … Additionally, a closed-loop color conversion module, such as closed-loop color conversion module 410, may receive attribute information about an original non-compressed point cloud, such as color values of points of the point cloud prior to being down-sampled, up-sampled, color converted, etc. … A closed-loop color conversion module may receive a compressed version of a point cloud … and also a reference version of the point cloud before any distortion has been introduced into the point cloud due to sampling, compression, or color conversion.” ¶ 0273).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to modify the method for rendering the image according to claim 2 and claim 14 of XIAO-2022*SIAO to include the disclosure that any one piece of raw point cloud data comprises color data and the disclosure that the raw point cloud data satisfying the filtering condition comprises raw point cloud data with color data matching a set color of MAMMOU. The motivation for this modification is to account for distortion caused by projecting a point cloud onto patches and packing the patches into image frames, and to account for distortion caused by video encoding and/or decoding the image frames comprising packed patches. To do this, a closed-loop color conversion module may take as an input a reference point cloud original color and a video compressed image frame comprising packed patches, wherein the packed patches of the image frame have been converted from a first color space to a second color space. The closed-loop color conversion module may decompress the compressed image frame using a video decoder and reconstruct the original point cloud using the decompressed image frames. The closed-loop color conversion module may then determine color values for points of the decompressed point cloud based on attribute and/or texture information included in the decompressed patches of the decompressed image frames (in the converted color space). The closed-loop color conversion module may then compare the down-sampled and up-sampled colors of the reconstructed point cloud to the colors of the original non-compressed point cloud. Based on this comparison, the closed-loop color conversion module may then adjust … parameter(s) used to convert the image frames from the original color space to the second color space, wherein the … parameter(s) are adjusted to improve quality of the final decompressed point cloud colors and to reduce the size of the compressed point cloud (MAMMOU; ¶ [0086]).
Regarding claim 13 and claim 17, XIAO-2022*SIAO disclose the method for rendering the image according to claim 2 and claim 14; however, XIAO-2022*SIAO do not explicitly disclose the following limitations, which MAMMOU discloses that:
the any one piece of raw point cloud data comprises reflection intensity data (MAMMOU; ¶ 0076-77; “… a system, may include … LIDAR system(s), 3-D camera(s), 3-D scanner(s), etc., and such sensor devices may capture spatial information, such as XYZ coordinates for points in a view of the sensor devices. … the spatial information may be relative to a local coordinate system or may be relative to a global coordinate system ([e.g.], a Cartesian coordinate system may have a fixed reference point, such as a fixed point on the earth, or may have a non-fixed local reference point, such as a sensor location). … such sensors may also capture attribute information for … point(s), such as color attributes, reflectivity attributes, velocity attributes, acceleration attributes, time attributes, modalities, and/or various other attributes. … other sensors, in addition to LIDAR systems, 3-D cameras, 3-D scanners, etc., may capture attribute information to be included in a point cloud.”); and
the raw point cloud data satisfying the filtering condition comprises raw point cloud data with reflection intensity data less than a set reflection intensity (MAMMOU; ¶ 360; “Point cloud data are associated with geometry information as well as other attributes, e.g. texture, color, reflectance information, etc. Improved performance [is] achieved by considering the relationships and characteristics across different attributes. … similarity/dissimilarity of the geometry sample values in the projected plane [is] accounted for when processing the corresponding samples in an attribute plane. … neighboring projected samples that correspond to the same/similar depth in the geometry plane are expected to be highly correlated. However, neighboring samples that have very dissimilar depth information are less likely to be correlated. Therefore, when processing such samples, depth information could also be considered to determine how these samples should be considered.” ¶ 362; “… other applications that utilize the proposed filter/processing techniques described above may include de-noising, de-banding, de-ringing, de-blocking, sharpening, edge enhancement, object extraction/segmentation, display mapping (e.g. for HDR applications), recoloring/tone mapping, etc. Such methods could also be utilized for quality evaluation, e.g. by pooling together and considering data (e.g. summing distortion values) in corresponding patches that also correspond to similar geometry information and other attributes when evaluating a particular distortion measurement. Processing may be purely spatial, e.g. only projected images that correspond to the same time stamp may be considered for such processing, however temporal/spatiotemporal processing may also be permitted, e.g. using motion compensated or motion adaptive processing strategies.”).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to modify the method for rendering the image according to claim 2 and claim 14 of XIAO-2022*SIAO to include the disclosure that the any one piece of raw point cloud data comprises reflection intensity data and the disclosure that the raw point cloud data satisfying the filtering condition comprises raw point cloud data with reflection intensity data less than a set reflection intensity of MAMMOU. The motivation for this modification is to compress 3-D data to support augmented- and virtual-reality applications. One such form of 3-D data includes point cloud representations, where objects are specified as a series of points that are described in terms of 3-D geometry and a set of attributes per point that may include information such as color, reflectance, time, etc. Compression of such information is highly desirable given the amount of space and bandwidth such data would require if not compressed (MAMMOU; ¶ [0363]).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Hackel et al., "Contour Detection in Unstructured 3D Point Clouds", published 2016, describe methods for determining contours using a binary classifier in 3-D point cloud data, particularly along lines, edges, and corners in the three-dimensional data to find polygonal elements in architectural façades.
Yun et al. (U.S. PG-PUB 2023/0350065) discloses a method/system for extracting boundary lines between tree crown in a tree canopy using airborne LiDAR data.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JONATHAN M COFINO whose telephone number is (303) 297-4268. The examiner can normally be reached Monday-Friday 10A-4P MT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kent Chang can be reached at 571-272-7667. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JONATHAN M COFINO/ Examiner, Art Unit 2614
/KENT W CHANG/ Supervisory Patent Examiner, Art Unit 2614
Prosecution Insights