DETAILED ACTION
Notice of Pre-AIA or AIA Status
Claims 1-20 are pending in this application.
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Specification
The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1, 8 and 15 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Mai et al. Method to Perform 3D Localization of Text in Shipboard Point Cloud Data Using Corresponding 2D Image January 2021 DOI:10.1109/ICCE48956.2021.9352083 Conference: 2020 IEEE Eighth International Conference on Communications and Electronics (ICCE), hereby referred to as “Mai”.
Consider Claims 1, 8 and 15.
Mai teaches:
1. (Currently Amended) A method comprising: by a computing system: / 8. (Currently Amended) A system comprising: a processor; and a non-transitory machine-readable medium comprising instructions that, when executed by the processor, cause a computing system to: / 15. (Currently Amended) A non-transitory machine-readable medium comprising instructions that, when executed by a processor, cause a computing system to; (Mai: page 433 abstract, JD object detection and localization have been major focuses for the computer vision community for the past several years. However, extracting information from a JD point cloud is often a more cumbersome and labor intensive task compared to just 2D images. 2D techniques are more mature and also have far more labelled training data. The contribution of this work is to leverage 2D computer vision techniques on a panorama image and use that to extract information from the JD point cloud, in the case where there is an existing correspondence between the panorama and the point cloud. Performance of the algorithm will be based on 2D object detection, and JD position and rotation of the object. The objects of interest are text placards called "bullseyes" that are found throughout US Navy ships. JD data of this type of environment is limited, impacting the ability of researchers to develop and test their algorithms. Another contribution of this work is making available a large corpus of shipboard LiDAR scan data from the museum ship USS Midway.)
1. accessing a panoramic point cloud image of a physical environment, wherein the panoramic point cloud image comprises location data for points in the panoramic point cloud image; / 8. access a panoramic point cloud image of a physical environment, wherein the panoramic point cloud image comprises location data for points in the panoramic point cloud image; / 15. access a panoramic point cloud image of a physical environment, wherein the panoramic point cloud image comprises location data for points in the panoramic point cloud image; (Mai: pages 434-435, B. Object detection on 3D Point Clouds 3D detection and localization techniques [6] heavily depend on geometric features of the object of interest . Early stages of methods involve significant feature engineering ( e.g. surface normal, border point detection, depth information from 2D). Vote3Dee [12] and Hough Voting [13] focus on feature voting in grids of 3D point cloud. Pointrcnn [11] first generates a small number of high quality 3D proposals, then transforms the pooled points of each to canonical coordinates. Although these methods often obtain good results, they require significant computational power to process directly on 3D point clouds. A hydrid approach was proposed by Frustum PointNet [10]: to first process a related 2D image. Their method uses 2D detection on the image to identify and isolate a 3D frustum ( akin to view cone) as the region of interest before searching the space with proposed bounding boxes. Fundamentally their technique reduces the search space, but is still very computationally intensive. Our approach aims to localize the object in 3D direct ly from the 2D technique, offering a significant computational speed up. Section III. PROPOSED METHOD A. Description and operation Consider a colorized 3D point cloud of single scan, corresponding 2D RGB panorama image from the same location, and a one-to-one mapping describing that correspondence between them which we term the "grid." Note that the grid allows construction of the 2D image from the 3D point cloud and vice versa if needed. Further, if starting with just a 3D point cloud, one can construct a 2D panorama and define the grid mapping. The key concept of the algorithm is to perform detection and localization on the 2D image instead of 3D point cloud and then transform into 3D space via grid mapping. To improve text detection and recognition, one can transform the warped panorama using equirectangular projection to cube faces. A high level pipeline for the method can be seen in Figure 4.)
1. transforming the panoramic point cloud image into an alternate representation that reduces distortion in the panoramic point cloud image; / 8. and transform the panoramic point cloud image into an alternate representation that reduces distortion in the panoramic point cloud image; / 15. and transform the panoramic point cloud image into an alternate representation that reduces distortion in the panoramic point cloud image; (Mai: pages 434-435, Section III. PROPOSED METHOD A. Description and operation Consider a colorized 3D point cloud of single scan, corresponding 2D RGB panorama image from the same location, and a one-to-one mapping describing that correspondence between them which we term the "grid." Note that the grid allows construction of the 2D image from the 3D point cloud and vice versa if needed. Further, if starting with just a 3D point cloud, one can construct a 2D panorama and define the grid mapping. The key concept of the algorithm is to perform detection and localization on the 2D image instead of 3D point cloud and then transform into 3D space via grid mapping. To improve text detection and recognition, one can transform the warped panorama using equirectangular projection to cube faces. A high level pipeline for the method can be seen in Figure 4.
PNG
media_image1.png
683
643
media_image1.png
Greyscale
)
1. performing an optical character recognition (OCR) process on the alternate representation to determine text in the panoramic point cloud image; / 8. perform an optical character recognition (OCR) process on the alternate representation to determine text in the panoramic point cloud image; / 15. perform an optical character recognition (OCR) process on the alternate representation to determine text in the panoramic point cloud image; (Mia: page 435 section III PROPOSED METHOD Next text detection , recognition, and 2D localization are performed using East Text Detection and Tesseract OCR with Long Term Short Term Memory (LSTM). East Text Detector will output location and orientation of boxes containing text. Warp perspective [16] straightens and crops all the rectangles. Tesseract then performs recognition. Unique features of our text of interest are chosen to remove false positives. Bullseyes all contain numbers, the symbol '-', and color channel > 100 (blue for modern and red for legacy). The detections that remain are assumed to be various sections of the same bullseye. If more than one, we choose the two nearest centers and average their positions to obtain the predicted bullseye center. For visualization, the prediction rectangle is constructed to be 400 pixels wide and 300 pixels high, based on initial observations.)
1. constructing text labels to track the text determined in the panoramic point cloud image; / 8. construct text labels to track the text determined in the panoramic point cloud image; / 15. construct text labels to track the text determined in the panoramic point cloud image; (Mai: page 436-437 section IV. EXPERIMENTS In this section, we will discuss the experiments and present the results from 3D text detection and localization of bullseyes on three ships, in order to validate the performance of the algorithm. Detection focuses on 2D performance whereas localization tests against 3D metrics. For detection, we focus on 2D detection and measure binary success or failure. Recall that a pipeline of East Detection, Tesseract OCR, and incorporating specific features was used to detect and 2D localize candidate text. We present a confusion matrix for each ship in Figures 5, 6, 7. Ground truth is established by manual visual inspection. For localization in 3D, we test both position and rotation to measure performance. Ground truth is established by manually labelling the center of bullseye in 2D grid coordinates (u,v) using SCENE [1 5]. The normal vector to the surface is obtained by picking three points to define a plane in Cloud Compare [17] and then rotating for visual confirmation. Figure 8 visualizes such an example. For Ship A, the ground truth selections are included alongside the raw data for purposes of reproducibility. Across all ships, an arbitrary subset of 50 scans (true positive) were selected for this analysis. Positional accuracy compares the Euclidean distance of center of the predicted box to the ground truth, as seen in Figure 9. Note that first the (u,v) grid points are mapped to their corresponding 3D value through grid mapping. Rotational accuracy compares the angle of the normal vectors of the prediction to the ground truth, as seen in Figure 10 . Note ground truth is established without concern for orientation, so the absolute value of the normalized dot product is used for the comparison.)
1. and supporting text searches for the physical environment through the text labels. / 8. and support text searches for the physical environment through the text labels. / 15. and support text searches for the physical environment through the text labels. (Mai: page 435 B. Advantages This approach circumvents the need for labelled training data by leveraging well developed pretrained 2D techniques - necessitated by lack of labelled 3D data. Computational complexity is less of a concern compared with purely 3D techniques. This means the point cloud does not need to be decimated to a more manageable size. Since the 3D localization is performed directly from detected 2D features, there is no 3D search. Recall Frustum PointNet [10] uses the 2D image to isolate a 3D frustum as the region of interest before searching the space with proposed bounding boxes. Fundamentally their technique reduces the search space. Performing our search on the 2D image which is in the low MBs is better than adding a dimension and working with a point cloud near 1GB. C. 2D Panorama Reconstruction from 3D point cloud The panorama and grid are outputs of SCENE. However, given just a 3D point cloud , one can construct both of these directly. This could be used to adapt this technique to datasets with only a point cloud or to virtually change perspective to a nonscanner location. Assume a global coordinate system (world coordinate system) origin at (0,0,0) and the camera pose with roll, pitch and yaw information. One can take the xyz coordinate in the world coordinate (Pw) and subtract from it the location of the camera (C) to obtain the body coordinate of the 3D point cloud. In order to obtain the camera coordinate system we need to obtain the rotation matrix (R) from the roll, pitch, and yaw. Page 438 In some cases, there is no associated point for that exact grid value (and the algorithm just returns the default (0, 0, 0) value. Both of these are fundamentally sensor issues, but relying on a single value leaves the algorithm at the mercy of that sensor limitation. East detection gives the location of the rectangle that contain text, and also the rotation angle of the text. The rotation angle directly affects the quality of the recognition task using Tesseract with straight lines giving a better result. Although, we used warp perspective to improve the engine, those reading errors are inevitable. Size of the rectangles also effects the reading quality. Zoomed in, Tesseract can read smaller more detailed text, while a larger rectangle lets it read the whole sentence. We chose a rectangle of 50 pixels, to capture the whole sentence based on initial observations. This choice might be causing errors in some cases. Then after obtaining the (u ,v) coordinate of the center, again based on initial observations, we chose a rectangle 500 wide and 400 pixels high to bound the bullseye. Depending on distance from the sensor, sometimes this choice is incorrectly sized for the bullseye in the 2D image. )
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are such that the subject matter as a whole would have been obvious at the time the invention was made to a person having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the manner in which the invention was made.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over Mai2 et al. (US PGPub US 2023/0206387, filed December 11, 2021, hereby referred to as “Mai2”), in view of Simek et al. (US PGPub US 2018/0139431, hereby referred to as “Simek”).
Consider Claims 1, 8 and 15.
Mai2 teaches:
1. (Currently Amended) A method comprising: by a computing system: / 8. (Currently Amended) A system comprising: a processor; and a non-transitory machine-readable medium comprising instructions that, when executed by the processor, cause a computing system to: / 15. (Currently Amended) A non-transitory machine-readable medium comprising instructions that, when executed by a processor, cause a computing system to; (Mai2: abstract, The present invention relates to a systems and methods to perform 3D localization of target objects in point cloud data using a corresponding 2D image. According to an illustrative embodiment of the present disclosure, a target environment is imaged with a camera to generate a 2D panorama and a scanner to generate a 3D point cloud. The 2D panorama is mapped to the point cloud with a 1 to 1 grid map. The target objects are detected and localized in 2D before being mapped back to the 3D point cloud. [0003]-[0005], Figures 1-2, [0015] FIG. 3 shows a flow diagram of an exemplary method. At step 101: providing a computer vision system comprising a camera and a scanner. At step 103: identifying a class of target objects in the operating environment. At step 105: scanning the operating environment with the scanner to collect distance return values. At step 107: scanning the operating environment with the camera to collect a 2D panoramic image. At step 109: detecting target objects within the 2D panoramic image. At step 111: mapping the distance return values to the 2D panoramic image. At step 113: projecting the 2D panoramic image onto a cube map. At step 115: 2D detection within individual boxes of the cube map. At step 117: mapping cube map to panorama. At step 119: mapping panorama to 3D point cloud. At step 121: 3D localization of target data.)
1. accessing a panoramic point cloud image of a physical environment, wherein the panoramic point cloud image comprises location data for points in the panoramic point cloud image; / 8. access a panoramic point cloud image of a physical environment, wherein the panoramic point cloud image comprises location data for points in the panoramic point cloud image; / 15. access a panoramic point cloud image of a physical environment, wherein the panoramic point cloud image comprises location data for points in the panoramic point cloud image; (Mai2: [0011] FIG. 1 shows a flow diagram of an exemplary localization method. In exemplary operating environments, users need to collect 3D data for purposes of configuration management of its assets and facilities in order to improve installations and maintenance by knowing where everything is located. In exemplary embodiments, a scanner (e.g. LIDAR) sits atop a tripod and collects distance returns over a full range of vision (e.g., a full sphere minus blind spots). A second pass of the scanner can then collect photo imagery and stitch them together to provide RGB color values to each point in the point cloud and a corresponding 2D panorama image. The resulting input data is a colorized 3D point cloud and associated 2D panorama image. In exemplary embodiments, the system uses this data to create a 3D localization of target objects. Target objects can be navigation markers (e.g., location labels, name plates, safety symbols, etc.) located throughout an operating environment (e.g. a facility or building). Target objects can also include text (e.g., specific text sequences or any text sequence matching a specific text format). If the environment is relatively featureless, many spaces may look the same, and it is very easy to lose track of which direction you are facing. Humans are easily disoriented, and standard machine computer vision algorithms may suffer from similar problems. The target objects are unique objects in the environment (e.g., informational text printed on walls or other surfaces) that can allow users to approximate their current location. Exemplary algorithms identify target objects in the scans and ultimately output the 3D coordinates for their location. Subsequently, these positions can be used as known navigation references for other computer vision algorithms, enabling robotics, augmented reality, and other applications without needing to artificially adding features (e.g. QR codes) to the environment. This circumvents an unnecessary initialization step which may limit or prevent the deployment of these technologies. While significant amounts of data of this form can be collected, it is often unlabeled, which can present a challenge for detection tasks using machine learning. To leverage or develop purely 3D techniques, a point cloud would need to be labelled which can be a time and resource intensive task to perform. In comparison, 2D computer vision techniques are already mature and models have been trained extensively on large labelled datasets. The sensor already outputs a 2D panorama image and further establishes a direct correspondence between the 2D data and the 3D point cloud. Exemplary embodiments build a complete pipeline that receives unlabeled sensor data and transforms it into detected, recognized, and localized target objects in the 3D point cloud.)
1. transforming the panoramic point cloud image into an alternate representation that reduces distortion in the panoramic point cloud image; / 8. and transform the panoramic point cloud image into an alternate representation that reduces distortion in the panoramic point cloud image; / 15. and transform the panoramic point cloud image into an alternate representation that reduces distortion in the panoramic point cloud image; (Mai2: [0012] In exemplary embodiments, the system generates a colorized 3D point cloud of single scan and a corresponding 2D RGB panorama image from the same location, and then the system one-to-one maps that correspondence between them to generate a grid. The grid allows construction of the 2D image from the 3D point cloud and vice versa if needed. Further, if starting with just a 3D point cloud, one can construct a 2D panorama and define the grid mapping. The key concept of the exemplary method is to perform detection and localization on the 2D image instead of 3D point cloud and then transform into 3D space via grid mapping. To improve text detection and recognition, one can transform the warped panorama using equirectangular projection to cube faces. Next target detection, recognition, and 2D localization are performed using detection and recognition software (e.g., text or image recognition). In exemplary embodiments, the target objects are predetermined strings of text. In these embodiments, the text detection software can output location and orientation of boxes containing text (e.g. East Text Detector, etc.). A perspective tool (e.g., Warp function in OpenCV, etc.) can be used to straighten and crop all of the rectangles for easier recognition. The text recognition software (e.g., Tesseract OCR with Long Term Short Term Memory (LSTM), etc.) can then perform text recognition of the detected text. A user can choose unique features of the text of interest to improve accuracy (e.g., remove false positives or negatives). The detections that remain can be assumed to be various sections of the same target. If more than one detection occurs, the system can choose the two nearest centers and average their positions to obtain the predicted bullseye center. For alternative exemplary embodiments, the system can be designed to detect and recognize multiple targets within a single scan. In these embodiments, clustering techniques can be used to separate or group targets.)
1. performing an optical character recognition (OCR) process on the alternate representation to determine text in the panoramic point cloud image; / 8. perform an optical character recognition (OCR) process on the alternate representation to determine text in the panoramic point cloud image; / 15. perform an optical character recognition (OCR) process on the alternate representation to determine text in the panoramic point cloud image; (Mai2: [0012] Next target detection, recognition, and 2D localization are performed using detection and recognition software (e.g., text or image recognition). In exemplary embodiments, the target objects are predetermined strings of text. In these embodiments, the text detection software can output location and orientation of boxes containing text (e.g. East Text Detector, etc.). A perspective tool (e.g., Warp function in OpenCV, etc.) can be used to straighten and crop all of the rectangles for easier recognition. The text recognition software (e.g., Tesseract OCR with Long Term Short Term Memory (LSTM), etc.) can then perform text recognition of the detected text. A user can choose unique features of the text of interest to improve accuracy (e.g., remove false positives or negatives). The detections that remain can be assumed to be various sections of the same target. If more than one detection occurs, the system can choose the two nearest centers and average their positions to obtain the predicted bullseye center. For alternative exemplary embodiments, the system can be designed to detect and recognize multiple targets within a single scan. In these embodiments, clustering techniques can be used to separate or group targets.)
1. constructing text labels to track the text determined in the panoramic point cloud image; / 8. construct text labels to track the text determined in the panoramic point cloud image; / 15. construct text labels to track the text determined in the panoramic point cloud image; (Mai2: [0013] Exemplary embodiments use the panorama constructed by the scanner’s 3D localizing software. While this gives a high quality image, it does so independent of the point cloud data. If a certain pixel does not have a corresponding 3D point associated with it, the algorithm assigns the point (0,0,0) as default. Next the system can calculate a surface normal vector at the center point. The text is on the surface of a wall and facing the scanner. Exemplary embodiments use the 5,000 nearest neighbors of the center for the calculation. Given the normal vector, to finish constructing the box, the system needs to choose a vector in the surface itself. Exemplary embodiments choose a surface vector that is as horizontally level as possible. Exemplary embodiments then construct a 3D box around the center along these vectors, with a predetermined width (e.g. 0.12 meters) along the normal vector and a predetermined length and height (0.6 meters) along the other two vectors. This approach circumvents the need for labelled training data by leveraging well developed pretrained 2D techniques. Compared with purely 3D techniques, the exemplary embodiments computational complexity is significantly lower and therefore less of a concern. As a result, the point cloud does not need to be decimated to a more manageable size. Because the 3D localization is performed directly from detected 2D features, a 3D search is not required. Performing the search on the 2D image which is comparatively small (megabyte scale) is more efficient than adding a dimension and working with a large point cloud (gigabyte scale).)
1. and supporting text searches for the physical environment through the text labels. / 8. and support text searches for the physical environment through the text labels. / 15. and support text searches for the physical environment through the text labels. (Mai2: [0014] FIG. 2 shows an exemplary representative of mapping between 3D and 2D through the grid. The panorama and grid are outputs of 3D localization software (e.g., SCENE). However, given just a 3D point cloud, one can construct both of these directly. This could be used to adapt this technique to datasets with only a point cloud or to virtually change perspective to a non-scanner location. Assume a global coordinate system (world coordinate system) origin at (0,0,0) and the camera pose with roll, pitch and yaw information. One can take the xyz coordinate in the world coordinate (Pw) and subtract from it the location of the camera (C) to obtain the body coordinate of the 3D point cloud. In order to obtain the camera coordinate (Pc) system we need to obtain the rotation matrix (R) from the roll, pitch, and yaw, as shown in Equations 1 and 2.
PNG
media_image2.png
323
564
media_image2.png
Greyscale
[0015] FIG. 3 shows a flow diagram of an exemplary method. At step 101: providing a computer vision system comprising a camera and a scanner. At step 103: identifying a class of target objects in the operating environment. At step 105: scanning the operating environment with the scanner to collect distance return values. At step 107: scanning the operating environment with the camera to collect a 2D panoramic image. At step 109: detecting target objects within the 2D panoramic image. At step 111: mapping the distance return values to the 2D panoramic image. At step 113: projecting the 2D panoramic image onto a cube map. At step 115: 2D detection within individual boxes of the cube map. At step 117: mapping cube map to panorama. At step 119: mapping panorama to 3D point cloud. At step 121: 3D localization of target data.)
Even if Mai2 does not specifically teach the term “distortion” from the limitation:
transform the panoramic point cloud image into an alternate representation that reduces distortion in the panoramic point cloud image
Simek teaches:
1. (Currently Amended) A method comprising: by a computing system: / 8. (Currently Amended) A system comprising: a processor; and a non-transitory machine-readable medium comprising instructions that, when executed by the processor, cause a computing system to: / 15. (Currently Amended) A non-transitory machine-readable medium comprising instructions that, when executed by a processor, cause a computing system to; (Simek: abstract, [0049] System 100 facilitates capturing and aligning panoramic image and depth data. In the embodiment shown, system 100 includes a 2D/3D panoramic capture device 102 that is configured to capture 2D and 3D panoramic imagery. In particular, the 2D/3D panoramic capture device 102 can include one or more color cameras that can capture 2D images that when combined, provide up to a 360° (horizontal) field-of-view of an environment. In some embodiments, the 2D/3D panoramic capture device 102 can include a plurality of color cameras whose collective fields-of-view span up to 360°, thereby allowing an entire panoramic image to be captured simultaneously and merged into a single panoramic image or video. In other embodiments, the 2D/3D panoramic capture device 102 can be configured to rotate about a fixed vertical axis and capture 2D images of an environment using one or more color cameras at different azimuth angles or orientations of rotation relative to a center point through which the vertical axis passes, wherein the collective fields-of-view of the combined 2D images can provide up to a 360° view of the environment. The azimuth function is a spatial numeric measurement that generates a value between 0 and 360 (degrees) that gives the orientation or angle of rotation of a feature. As used herein, the azimuth is measured as the degrees of clockwise rotation from the positive y axis. In other words, with respect to lines provided on the same plane, the azimuth for a line pointing forward is 0°, a line pointing right is 90°, a line pointing backwards is 180°, and a line pointing left is 270°. [0050] The 2D/3D panoramic capture device 102 can further include one or more depth sensor devices that can capture or sense depth information for visual features included in the 2D images. These depth sensor devices can include but are not limited to: time-of-flight sensor devices, structured light sensor devices, light detection and ranging (LiDAR) devices, assisted stereo devices, and passive stereo devices. For example, in some embodiments, the 2D/3D panoramic capture device 102 can include a plurality of depth sensor devices whose collective fields-of-view span up to 360°, thereby allowing an entire panoramic depth map to be captured simultaneously and merged into a single panoramic depth map for a corresponding panoramic 2D image. In other embodiments, the 2D/3D panoramic capture device 102 can be configured to rotate about a fixed vertical axis and capture 3D depth data of an environment using one or more depth sensor devices at different azimuth angles of rotation relative to the center point, wherein the collective fields-of-view of the combined 3D depth data provides a depth map of the environment that spans up to 360°. In other embodiments, the 2D/3D panoramic capture device 102 can be configured to generate stereo images or images with partially overlapping fields-of-view from which depth information can be extracted using passive stereo depth derivation techniques, active stereo depth derivation techniques, and/or machine learning based derivation techniques for depth estimation. [0051] System 100 further includes a user device 106 and optionally a 3D modeling and navigation server device 112. In various embodiments, the user device 106 and/or the 3D modeling and navigation server device 112 can facilitate various aspects of the capture process. The user device 106 and/or the 3D modeling and navigation server device 112 can also facilitate processing of the 3D panoramic imagery captured by the 2D/3D panoramic capture device 102. [0052] In one embodiment, the user device 106 can include a personal computing device (e.g. a tablet computer, laptop computer, a smartphone, etc.) that can be communicatively coupled to the 2D/3D panoramic capture device 102 and provide a control user interface that facilitates operation of the 2D/3D panoramic capture device 102 in association with the capture process. For example, the user device 106 can receive user input via the control user interface that controls one or more features and functionalities of the 2D/3D panoramic capture device 102.)
1. accessing a panoramic point cloud image of a physical environment, wherein the panoramic point cloud image comprises location data for points in the panoramic point cloud image; / 8. access a panoramic point cloud image of a physical environment, wherein the panoramic point cloud image comprises location data for points in the panoramic point cloud image; / 15. access a panoramic point cloud image of a physical environment, wherein the panoramic point cloud image comprises location data for points in the panoramic point cloud image; (Simek: [0095] FIG. 5 presents a schematic block diagram of example processing component 420 in accordance with various aspects and embodiments described herein. In various embodiments, the processing component 420 can include 2D/3D panoramic image generation component 502, to facilitate generating panoramic 2D images, panoramic 3D images and 3D models of an environment. In various embodiments, the 2D/3D panoramic image generation component 502 can facilitate generating panoramic 2D images, panoramic 3D images, and 3D models in association with capture of data (e.g., 2D image data, video data, depth data, movement data, positional/location data, etc.) by a 2D/3D panoramic capture device. In this regard, the 2D/3D panoramic image generation component 502 can facilitate generating panoramic 2D images, panoramic 3D images, and 3D models as the cameras (e.g., cameras 206), the depth detection components (e.g., depth detection component 204) and other potential input devices are regularly or continuously capturing and inputting data. The 2D/3D panoramic image generation component 502 can also facilitate generating panoramic 2D images, panoramic 3D images and 3D models in a static environment. In this regard, the 2D/3D panoramic image generation component 502 can facilitate generating panoramic 2D images, panoramic 3D images and 3D models after a capture process has been completed and the 2D/3D panoramic capture device no longer captures and provides any additional data (e.g., 2D image data, video data, depth data, movement data, positional/location data, etc). The 2D/3D panoramic image generation component 502 can include 2D/3D aggregation component 504, projection component 506, stitching component 508, depth data optimization component 510, stereo depth derivation component 512, object removal component 514, and panoramic video generation component 516. In some embodiments, the processing component 420 can further include 3D model generation component 518, rendering component 520, and positioning component 522. Repetitive description of like elements employed in respective embodiments is omitted for sake of brevity. [0096] The 2D/3D panoramic image generation component 502 can be configured to efficiently (e.g. in real-time or substantially real-time) merge and align 2D images and 3D depth data captured by the 2D/3D panoramic capture device 400, (and other 2D/3D panoramic capture devices described herein), to generate a 2D panoramic image and/or a 3D panoramic depth image (e.g. a 3D depth map or model). In some implementations, the 2D/3D panoramic image generation component 502 can further merge and align 2D images and 3D data captured by the 2D/3D panoramic capture device 400 (and other 2D/3D panoramic capture devices described herein), to facilitate generating 2D panoramic video and/or a 3D panoramic video.)
1. transforming the panoramic point cloud image into an alternate representation that reduces distortion in the panoramic point cloud image; / 8. and transform the panoramic point cloud image into an alternate representation that reduces distortion in the panoramic point cloud image; / 15. and transform the panoramic point cloud image into an alternate representation that reduces distortion in the panoramic point cloud image; (Simek: In some embodiments, the 3D depth data, (including the 3D depth data projected by projection component 506), can include 3D depth data obtained from different sensor and/or depth derivation modalities having different strengths and weaknesses. For example, time-of-flight sensor devices are capable of generating 3D depth data for surfaces with uniform color where passive stereo fails. However, 3D data determined using passive stereo techniques can provide depth measurements for a longer range relative to time-of-flight based 3D depth data, has higher resolution, is not prone to temporal noise, and is not susceptible to distortions like multipath interference. Thus, in various embodiments, the 2D/3D panoramic capture devices described herein can employ two or more different types of depth sensor devices and/or depth derivation modalities to capture and/or generate depth data. The different modalities can include but are not limited to: time-of-flight based depth detection, structured light based depth detection, LiDAR based depth detection, light/laser assisted or active stereo based depth detection, and passive stereo based depth detection. For example, in various embodiments, the one or more depth detection components 204 of 2D/3D panoramic capture device 400 can include one or more time-of-flight sensor devices and the one or more depth detection components 204 can generate and/or determine time-of-flight based depth information. In another example, the one or more depth detection components 204 of 2D/3D panoramic capture device 400 can capture and determine 3D data from one or more structured light sensor devices and/or one or more LiDAR sensor devices. The 2D/3D panoramic capture device 400 can further employ an active stereo system, wherein the one or more depth detection components 204 include a light projection unit configured to project some form of light during capture of stereo images via stereo cameras included in the depth detection components 204 and/or via pairs of cameras 206. Still in other embodiment, the 2D/3D panoramic capture device 400 can generate stereo images and determine depth information using passive stereo processing functions. [0102] The depth data optimization component 510 can be configured to analyze 3D depth data obtained from different sensor and/or depth derivation modalities to determine an optimized unified interpretation of the depth data. In particular, the depth data optimization component 510 can analyze different types of depth data captured and/or determined using different types of depths sensor devices and/or depth derivation techniques to determine optimized spatial coordinates for 3D points collectively represented by the depth data. For example, the depth data optimization component 510 can combine the different types of depth data associated with the same area, volume, cell or 3D point to determine an average or optimized 3D spatial position for the area, cell, volume or 3D point. For instance, the depth data captured by a 2D/3D panoramic capture device can include sets of different types of depth data respectively captured by different types of depth sensors devices and/or determined using different types of depth derivation techniques (e.g. passive stereo depth derivation techniques, active stereo depth derivation techniques, etc.). In one implementation, the depth data optimization component 510 can be configured to combine subsets of the different sets of depth data associated with the same three-dimensional volume to determine spatial positions for points included in the same three-dimensional volume. In an aspect, the depth data optimization component 510 can employ a heuristic to evaluate the quality of 3D depth data captured of the same space from the same location and with different depth detection modalities to determine a unified interpretation of the depth data.)
It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to modify Mai’s method and system for 3D localization of target objects using point cloud data mapped with panoramic images to leverage Simek’s method and system for aligning panoramic and depth data that addresses the possibility of distortion and ensures more accurate visually blended 360-degree interpolated panoramic image data. The determination of obviousness is predicated upon the following findings: Both Mai and Simek are directed towards the same field of endeavor, and one skilled in the art would have been motivated to modify Mai in order to ensure more photorealistic panoramic 2D images captured with 3D depth information accurately. Furthermore, the prior art collectively includes each element claimed (though not all in the same reference), and one of ordinary skill in the art could have combined the elements in the manner explained above using known engineering design, interface and programming techniques, without changing a “fundamental” operating principle of Mai, while the teaching of Simek continues to perform the same function as originally taught prior to being combined, in order to produce the repeatable and predictable result of more efficiently aligning the 2D images to generate immersive 3D environments accurately. It is for at least the aforementioned reasons that the examiner has reached a conclusion of obviousness with respect to the claim in question.
Consider Claims 2 and 9.
The combination of Mai2 and Simek teaches:
2. (Currently Amended) The method of claim 1, comprising constructing the text labels to include, for a given text section that includes a given text identified in the panoramic point cloud image: an identifier for the panoramic point cloud image; the location data for a selected point in the text section; and the given text included in the given text section. / 9. (Currently Amended) The system of claim 8, wherein the instructions, when executed, cause the computing system to the text labels to include, for a given text section that includes a given text identified in the panoramic point cloud image: an identifier for the panoramic point cloud image; the location data for a selected point in the text section; and the given text included in the given text section. Mai2: [0014] FIG. 2 shows an exemplary representative of mapping between 3D and 2D through the grid. The panorama and grid are outputs of 3D localization software (e.g., SCENE). However, given just a 3D point cloud, one can construct both of these directly. This could be used to adapt this technique to datasets with only a point cloud or to virtually change perspective to a non-scanner location. Assume a global coordinate system (world coordinate system) origin at (0,0,0) and the camera pose with roll, pitch and yaw information. One can take the xyz coordinate in the world coordinate (Pw) and subtract from it the location of the camera (C) to obtain the body coordinate of the 3D point cloud. In order to obtain the camera coordinate (Pc) system we need to obtain the rotation matrix (R) from the roll, pitch, and yaw, as shown in Equations 1 and 2.
PNG
media_image2.png
323
564
media_image2.png
Greyscale
[0015] FIG. 3 shows a flow diagram of an exemplary method. At step 101: providing a computer vision system comprising a camera and a scanner. At step 103: identifying a class of target objects in the operating environment. At step 105: scanning the operating environment with the scanner to collect distance return values. At step 107: scanning the operating environment with the camera to collect a 2D panoramic image. At step 109: detecting target objects within the 2D panoramic image. At step 111: mapping the distance return values to the 2D panoramic image. At step 113: projecting the 2D panoramic image onto a cube map. At step 115: 2D detection within individual boxes of the cube map. At step 117: mapping cube map to panorama. At step 119: mapping panorama to 3D point cloud. At step 121: 3D localization of target data.)
Consider Claims 3 and 10.
The combination of Mai2 and Simek teaches:
3. (Currently Amended) The method of claim 2, wherein the given text section comprises a bounding box for the given text in the panoramic point cloud image and the location data is for a selected point on or within the bounding box. / 10. (Currently Amended) The system of claim 9 wherein the given text section comprises a bounding box for the given text in the panoramic point cloud image and the location data is for a selected point on or within the bounding box. Mai2: [0014] FIG. 2 shows an exemplary representative of mapping between 3D and 2D through the grid. The panorama and grid are outputs of 3D localization software (e.g., SCENE). However, given just a 3D point cloud, one can construct both of these directly. This could be used to adapt this technique to datasets with only a point cloud or to virtually change perspective to a non-scanner location. Assume a global coordinate system (world coordinate system) origin at (0,0,0) and the camera pose with roll, pitch and yaw information. One can take the xyz coordinate in the world coordinate (Pw) and subtract from it the location of the camera (C) to obtain the body coordinate of the 3D point cloud. In order to obtain the camera coordinate (Pc) system we need to obtain the rotation matrix (R) from the roll, pitch, and yaw, as shown in Equations 1 and 2.
PNG
media_image2.png
323
564
media_image2.png
Greyscale
[0015] FIG. 3 shows a flow diagram of an exemplary method. At step 101: providing a computer vision system comprising a camera and a scanner. At step 103: identifying a class of target objects in the operating environment. At step 105: scanning the operating environment with the scanner to collect distance return values. At step 107: scanning the operating environment with the camera to collect a 2D panoramic image. At step 109: detecting target objects within the 2D panoramic image. At step 111: mapping the distance return values to the 2D panoramic image. At step 113: projecting the 2D panoramic image onto a cube map. At step 115: 2D detection within individual boxes of the cube map. At step 117: mapping cube map to panorama. At step 119: mapping panorama to 3D point cloud. At step 121: 3D localization of target data.)
Consider Claims 4, 11 and 17.
The combination of Mai2 and Simek teaches:
4. (Currently Amended) The method of claim 1, wherein the alternate representation comprises a cube map and wherein the text labels comprise spherical theta and phi angles for coordinates of selected locations in text sections in the panoramic point cloud image that include the text. / 11. (Currently Amended) The system (100) of claim 8, wherein the alternate representation comprises a cube map and wherein the text labels comprise spherical theta and phi angles for coordinates of selected locations in text sections in the panoramic point cloud image that include the text. / 17. (New) The non-transitory machine-readable medium of claim 15, wherein the alternate representation comprises a cube map and wherein the text labels comprise spherical theta and phi angles for coordinates of selected locations in text sections in the panoramic point cloud image that include the text. (Mai2: [0014] FIG. 2 shows an exemplary representative of mapping between 3D and 2D through the grid. The panorama and grid are outputs of 3D localization software (e.g., SCENE). However, given just a 3D point cloud, one can construct both of these directly. This could be used to adapt this technique to datasets with only a point cloud or to virtually change perspective to a non-scanner location. Assume a global coordinate system (world coordinate system) origin at (0,0,0) and the camera pose with roll, pitch and yaw information. One can take the xyz coordinate in the world coordinate (Pw) and subtract from it the location of the camera (C) to obtain the body coordinate of the 3D point cloud. In order to obtain the camera coordinate (Pc) system we need to obtain the rotation matrix (R) from the roll, pitch, and yaw, as shown in Equations 1 and 2.
PNG
media_image2.png
323
564
media_image2.png
Greyscale
[0015] FIG. 3 shows a flow diagram of an exemplary method. At step 101: providing a computer vision system comprising a camera and a scanner. At step 103: identifying a class of target objects in the operating environment. At step 105: scanning the operating environment with the scanner to collect distance return values. At step 107: scanning the operating environment with the camera to collect a 2D panoramic image. At step 109: detecting target objects within the 2D panoramic image. At step 111: mapping the distance return values to the 2D panoramic image. At step 113: projecting the 2D panoramic image onto a cube map. At step 115: 2D detection within individual boxes of the cube map. At step 117: mapping cube map to panorama. At step 119: mapping panorama to 3D point cloud. At step 121: 3D localization of target data.)
Consider Claims 5, 12 and 18.
The combination of Mai2 and Simek teaches:
5. (Currently Amended) The method of claim 1, wherein supporting text searches for the physical environment through the text labels comprises: identifying a text search term provided through a search query; and providing an oriented view of the physical environment that comprises the text search term, wherein the oriented view comprises a virtual view of the physical environment oriented with respect to a location in the physical environment that comprises the text search term. / 12. (Currently Amended) The system of claim 8, wherein the instructions when executed, cause the computing system to support text searches for the physical environment through the text labels by: identifying a text search term provided through a search query; and providing an oriented view of the physical environment that comprises the text search term, wherein the oriented view comprises a virtual view of the physical environment oriented with respect to a location in the physical environment that comprises the text search term. / 18. (New) The non-transitory machine-readable medium of claim 15, wherein the instructions, when executed, cause the computing system to support text searches for the physical environment through the text labels by: identifying a text search term provided through a search query; and providing an oriented view of the physical environment that comprises the text search term, wherein the oriented view comprises a virtual view of the physical environment oriented with respect to a location in the physical environment that comprises the text search term. (Mai2: [0014] FIG. 2 shows an exemplary representative of mapping between 3D and 2D through the grid. The panorama and grid are outputs of 3D localization software (e.g., SCENE). However, given just a 3D point cloud, one can construct both of these directly. This could be used to adapt this technique to datasets with only a point cloud or to virtually change perspective to a non-scanner location. Assume a global coordinate system (world coordinate system) origin at (0,0,0) and the camera pose with roll, pitch and yaw information. One can take the xyz coordinate in the world coordinate (Pw) and subtract from it the location of the camera (C) to obtain the body coordinate of the 3D point cloud. In order to obtain the camera coordinate (Pc) system we need to obtain the rotation matrix (R) from the roll, pitch, and yaw, as shown in Equations 1 and 2.
PNG
media_image2.png
323
564
media_image2.png
Greyscale
[0015] FIG. 3 shows a flow diagram of an exemplary method. At step 101: providing a computer vision system comprising a camera and a scanner. At step 103: identifying a class of target objects in the operating environment. At step 105: scanning the operating environment with the scanner to collect distance return values. At step 107: scanning the operating environment with the camera to collect a 2D panoramic image. At step 109: detecting target objects within the 2D panoramic image. At step 111: mapping the distance return values to the 2D panoramic image. At step 113: projecting the 2D panoramic image onto a cube map. At step 115: 2D detection within individual boxes of the cube map. At step 117: mapping cube map to panorama. At step 119: mapping panorama to 3D point cloud. At step 121: 3D localization of target data.)
Consider Claims 6, 13 and 19.
The combination of Mai2 and Simek teaches:
6. (Currently Amended) The method of claim 5, further comprising de-skewing other text sections in the oriented view that do not include the text search term. / 13. (Currently Amended) The system of claim 12, wherein the instructions, when executed, further cause the computing system to de-skew other text sections in the oriented view that do not include the text search term. / 19. (New) The non-transitory machine-readable medium of claim 18, wherein the instructions, when executed, further cause the computing system to de-skew other text sections in the oriented view that do not include the text search term. (Mai2: [0011], Figure 1; Target objects can be navigation markers (e.g., location labels, name plates, safety symbols, etc.) located throughout an operating environment (e.g. a facility or building). Target objects can also include text (e.g., specific text sequences or any text sequence matching a specific text format). If the environment is relatively featureless, many spaces may look the same, and it is very easy to lose track of which direction you are facing. Humans are easily disoriented, and standard machine computer vision algorithms may suffer from similar problems. The target objects are unique objects in the environment (e.g., informational text printed on walls or other surfaces) that can allow users to approximate their current location. Exemplary algorithms identify target objects in the scans and ultimately output the 3D coordinates for their location. Subsequently, these positions can be used as known navigation references for other computer vision algorithms, enabling robotics, augmented reality, and other applications without needing to artificially adding features (e.g. QR codes) to the environment. This circumvents an unnecessary initialization step which may limit or prevent the deployment of these technologies. While significant amounts of data of this form can be collected, it is often unlabeled, which can present a challenge for detection tasks using machine learning. To leverage or develop purely 3D techniques, a point cloud would need to be labelled which can be a time and resource intensive task to perform. Simek: [0060] In some embodiments, spatial metadata or tags including information about different objects or elements of the 3D space model can be applied to the 3D space model and also retained at the 3D modeling and navigation server device. For example, the tags can include text, images, audio, video, hyperlinks, etc., that can be represented by a tag icon that is spatially aligned in the 3D space model. Interaction with the tag icon as included in a rendered representation of the 3D space model can cause the server device to stream or otherwise provide the tag data/metadata to the user in a pop-up display window, a side panel, as a 2D or 3D object inside the 3D model, as a 2D overlay to the 3D model, or other suitable visual and/or audible form. [0061] In accordance with one or more embodiments, the 3D modeling and navigation server device 112 and the user device 106 can be configured to operate in client/server relationship, wherein the 3D modeling and navigation server device 112 provides the user device 106 access to 3D modeling and navigation services via a network accessible platform (e.g. a website, a thin client application, etc.) using a browser or the like. However, system 100 is not limited to this architectural configuration. For example, in some embodiments, one or more features, functionalities and associated components of the 3D modeling and navigation server device 112 can be provided on the user device 106 and/or the 2D/3D panoramic capture device 102, and vice versa. In another embodiment, the features and functionalities of the 2D/3D panoramic capture device 102, the user device 106 and the 3D modeling and navigation server device 112 can be provided on a single device. Further, the 3D modeling and navigation server device 112 can include any suitable device and is not limited to a device that operates as a “server” in a server/client relationship.)
Consider Claims 7, 14 and 20.
The combination of Mai2 and Simek teaches:
7. (Currently Amended) The method of claim 1, wherein supporting text searches for the physical environment through the text labels comprises: determining all text that is present in a given view of the physical environment; de-skewing text sections that include the text present in the given view of the physical environment; and presenting the de-skewed text sections with the text present in the given view of the physical environment. / 14. (Currently Amended) The system of claim 8, wherein the instructions, when executed, cause the computing system to support text searches for the physical environment through the text labels by: determining all text that is present in a given view of the physical environment; de-skewing text sections that include the text present in the given view of the physical environment; and presenting the de-skewed text sections with the text present in the given view of the physical environment. / 20. (New) The non-transitory machine-readable medium of claim 15, wherein the instructions, when executed, cause the computing system to support text searches for the physical environment through the text labels by: determining all text that is present in a given view of the physical environment; de-skewing text sections that include the text present in the given view of the physical environment; and presenting the de-skewed text sections with the text present in the given view of the physical environment. (Mai2: [0011], Figure 1; Target objects can be navigation markers (e.g., location labels, name plates, safety symbols, etc.) located throughout an operating environment (e.g. a facility or building). Target objects can also include text (e.g., specific text sequences or any text sequence matching a specific text format). If the environment is relatively featureless, many spaces may look the same, and it is very easy to lose track of which direction you are facing. Humans are easily disoriented, and standard machine computer vision algorithms may suffer from similar problems. The target objects are unique objects in the environment (e.g., informational text printed on walls or other surfaces) that can allow users to approximate their current location. Exemplary algorithms identify target objects in the scans and ultimately output the 3D coordinates for their location. Subsequently, these positions can be used as known navigation references for other computer vision algorithms, enabling robotics, augmented reality, and other applications without needing to artificially adding features (e.g. QR codes) to the environment. This circumvents an unnecessary initialization step which may limit or prevent the deployment of these technologies. While significant amounts of data of this form can be collected, it is often unlabeled, which can present a challenge for detection tasks using machine learning. To leverage or develop purely 3D techniques, a point cloud would need to be labelled which can be a time and resource intensive task to perform. Simek: [0060] In some embodiments, spatial metadata or tags including information about different objects or elements of the 3D space model can be applied to the 3D space model and also retained at the 3D modeling and navigation server device. For example, the tags can include text, images, audio, video, hyperlinks, etc., that can be represented by a tag icon that is spatially aligned in the 3D space model. Interaction with the tag icon as included in a rendered representation of the 3D space model can cause the server device to stream or otherwise provide the tag data/metadata to the user in a pop-up display window, a side panel, as a 2D or 3D object inside the 3D model, as a 2D overlay to the 3D model, or other suitable visual and/or audible form. [0061] In accordance with one or more embodiments, the 3D modeling and navigation server device 112 and the user device 106 can be configured to operate in client/server relationship, wherein the 3D modeling and navigation server device 112 provides the user device 106 access to 3D modeling and navigation services via a network accessible platform (e.g. a website, a thin client application, etc.) using a browser or the like. However, system 100 is not limited to this architectural configuration. For example, in some embodiments, one or more features, functionalities and associated components of the 3D modeling and navigation server device 112 can be provided on the user device 106 and/or the 2D/3D panoramic capture device 102, and vice versa. In another embodiment, the features and functionalities of the 2D/3D panoramic capture device 102, the user device 106 and the 3D modeling and navigation server device 112 can be provided on a single device. Further, the 3D modeling and navigation server device 112 can include any suitable device and is not limited to a device that operates as a “server” in a server/client relationship.)
Consider Claim 16.
The combination of Mai2 and Simek teaches:16. (New) The non-transitory machine-readable medium of claim 15, wherein the instructions, when executed, cause the computing system to construct the text labels to include, for a given text section that includes a given text identified in the panoramic point cloud image: an identifier for the panoramic point cloud image; the location data for a selected point in the text section; and the given text included in the given text section, wherein the given text section comprises a bounding box for the given text in the panoramic point cloud image and the location data is for a selected point on or within the bounding box. (Mai2: [0011], Figure 1; Target objects can be navigation markers (e.g., location labels, name plates, safety symbols, etc.) located throughout an operating environment (e.g. a facility or building). Target objects can also include text (e.g., specific text sequences or any text sequence matching a specific text format). If the environment is relatively featureless, many spaces may look the same, and it is very easy to lose track of which direction you are facing. Humans are easily disoriented, and standard machine computer vision algorithms may suffer from similar problems. The target objects are unique objects in the environment (e.g., informational text printed on walls or other surfaces) that can allow users to approximate their current location. Exemplary algorithms identify target objects in the scans and ultimately output the 3D coordinates for their location. Subsequently, these positions can be used as known navigation references for other computer vision algorithms, enabling robotics, augmented reality, and other applications without needing to artificially adding features (e.g. QR codes) to the environment. This circumvents an unnecessary initialization step which may limit or prevent the deployment of these technologies. While significant amounts of data of this form can be collected, it is often unlabeled, which can present a challenge for detection tasks using machine learning. To leverage or develop purely 3D techniques, a point cloud would need to be labelled which can be a time and resource intensive task to perform. Mai2: [0014] FIG. 2 shows an exemplary representative of mapping between 3D and 2D through the grid. The panorama and grid are outputs of 3D localization software (e.g., SCENE). However, given just a 3D point cloud, one can construct both of these directly. This could be used to adapt this technique to datasets with only a point cloud or to virtually change perspective to a non-scanner location. Assume a global coordinate system (world coordinate system) origin at (0,0,0) and the camera pose with roll, pitch and yaw information. One can take the xyz coordinate in the world coordinate (Pw) and subtract from it the location of the camera (C) to obtain the body coordinate of the 3D point cloud. In order to obtain the camera coordinate (Pc) system we need to obtain the rotation matrix (R) from the roll, pitch, and yaw, as shown in Equations 1 and 2.
PNG
media_image2.png
323
564
media_image2.png
Greyscale
[0015] FIG. 3 shows a flow diagram of an exemplary method. At step 101: providing a computer vision system comprising a camera and a scanner. At step 103: identifying a class of target objects in the operating environment. At step 105: scanning the operating environment with the scanner to collect distance return values. At step 107: scanning the operating environment with the camera to collect a 2D panoramic image. At step 109: detecting target objects within the 2D panoramic image. At step 111: mapping the distance return values to the 2D panoramic image. At step 113: projecting the 2D panoramic image onto a cube map. At step 115: 2D detection within individual boxes of the cube map. At step 117: mapping cube map to panorama. At step 119: mapping panorama to 3D point cloud. At step 121: 3D localization of target data. Simek: [0060] In some embodiments, spatial metadata or tags including information about different objects or elements of the 3D space model can be applied to the 3D space model and also retained at the 3D modeling and navigation server device. For example, the tags can include text, images, audio, video, hyperlinks, etc., that can be represented by a tag icon that is spatially aligned in the 3D space model. Interaction with the tag icon as included in a rendered representation of the 3D space model can cause the server device to stream or otherwise provide the tag data/metadata to the user in a pop-up display window, a side panel, as a 2D or 3D object inside the 3D model, as a 2D overlay to the 3D model, or other suitable visual and/or audible form. [0061] In accordance with one or more embodiments, the 3D modeling and navigation server device 112 and the user device 106 can be configured to operate in client/server relationship, wherein the 3D modeling and navigation server device 112 provides the user device 106 access to 3D modeling and navigation services via a network accessible platform (e.g. a website, a thin client application, etc.) using a browser or the like. However, system 100 is not limited to this architectural configuration. For example, in some embodiments, one or more features, functionalities and associated components of the 3D modeling and navigation server device 112 can be provided on the user device 106 and/or the 2D/3D panoramic capture device 102, and vice versa. In another embodiment, the features and functionalities of the 2D/3D panoramic capture device 102, the user device 106 and the 3D modeling and navigation server device 112 can be provided on a single device. Further, the 3D modeling and navigation server device 112 can include any suitable device and is not limited to a device that operates as a “server” in a server/client relationship.)
Conclusion
The prior art made of record in form PTO-892 and not relied upon is considered pertinent to applicant's disclosure.
PNG
media_image3.png
342
922
media_image3.png
Greyscale
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TAHMINA ANSARI whose telephone number is 571-270-3379. The examiner can normally be reached on IFP Flex - Monday through Friday 9 to 5.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, O’NEAL MISTRY can be reached on 313-446-4912. The fax phone numbers for the organization where this application or proceeding is assigned are 571-273-8300 for regular communications and 571-273-8300 for After Final communications. TC 2600’s customer service number is 571-272-2600.
Any inquiry of a general nature or relating to the status of this application or proceeding should be directed to the receptionist whose telephone number is 571-272-2600.
2674
/Tahmina Ansari/
June 23, 2026
/TAHMINA N ANSARI/Primary Examiner, Art Unit 2674
Prosecution Insights