Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2 and 12-15 are rejected under 35 U.S.C. 103 as being unpatentable over Y. Lee et al. (US 20220283310 A1; hereinafter Y. Lee '310) in view of H. Lee et al. (KR 20230112296 A; hereinafter H. Lee '296) in further view of Ebrahimi Afrouzi et al. (US 20210089040 A1; hereinafter Ebrahimi Afrouzi).
Regarding claim 14, Y. Lee '310 teaches an electronic device for generating a spatial map (“Further, the present invention provides a three-dimensional (3D) map generating system, including: a collecting device having a lidar sensor and a camera sensor; and a mapping device configured to generate a 3D feature map for a specific space, by using the collecting device, wherein the collecting device collects each of spatial data and image data on the specific space, by using the lidar sensor and the camera sensor, and wherein the mapping device estimates a movement trajectory of the lidar sensor by using the spatial data, and generates 3D structure data on the specific space based on Structure-From-Motion (SFM), by using the image data and the movement trajectory as input data,” (page 2, para [0021]).
“Meanwhile, the collecting device 110 is configured to perform SLAM when collecting data by using the sensing unit 112. That is, the collecting device 110 is configured to measure its location for the current time, and to make a map of the environment. SLAM may be performed by a processor of the collecting device 110, or by a controller 121 of the mapping device 120,” (page 4, para [0070]).
The electronic device includes the mapping device, controller, and / or collecting device. The spatial map includes a generated 3D map.),
the electronic device comprising: memory storing a datasets for generating a spatial map (“Additionally, the controller 121 controls overall operations of the collecting device 110…. As an example of data processing for SLAM, the controller 121 may perform simultaneous location estimation and map generation based on sensor data, and may estimate a movement path of the collecting device 110. For instance, the controller 121 may detect loop closure based on data sensed from the collecting device 110, generate location data based on the detected loop closure, and generate moving path information, and indoor map information of a target building based on the location data… In this case, the controller 121 controls a series of processes to generate, store, and update the aforementioned datasets by interworking with a storage unit 123,” (page 4, para [0070]-[0076]). The memory includes the storage unit 123.); and
at least one processor, wherein, by executing the program stored in the memory, the at least one processor is configured to (“SLAM may be performed by a processor of the collecting device 110, or by a controller 121 of the mapping device 120,” (page 4, para [0070]).):
obtain a captured image of a target area in a space (“…the 3D map generating method of the present invention includes: collecting spatial data and image data on a specific space by using a lidar sensor and a camera sensor each provided at a collecting device…,” (page 1, para [0012]).
“In the present invention, an “image” refers to, for example, a photo or a picture captured by a camera sensor, etc. For instance, after the mobile device captures an image by the camera sensor while moving at a specific space, a position of the mobile device may be estimated in real time through a process of comparing the captured image with 3D map data,” (page 3, para [0043]). The target area in a space includes Lee’s specific space.);
obtain light detection and ranging (LiDAR) scan data by scanning a depth of the target area with respect to a first height (“…the 3D map generating method of the present invention includes: collecting spatial data and image data on a specific space by using a lidar sensor and a camera sensor each provided at a collecting device…,” (page 1, para [0012]).
“The lidar sensor 112a is a sensor for obtaining 3D information on a surrounding space… The laser scanner may be a device for precisely depicting a surrounding area by irradiating laser pulses, receiving the irradiated laser pulses reflected from a surrounding object, and thereby measuring a distance to the object,” (page 3, para [0059]-[0062]).
“The spatial data may be point group data or scan data,” (page 5, para [0086]).
“…a second lidar sensor 112c may be horizontally arranged between the upper end and lower end of the collecting device 110 (e.g., a middle region). In this case, the first lidar sensor 112b in the vertical direction forms 3D point clouds of a high-density by a push-broom scanning scheme, and the second lidar sensor 112c in the horizontal direction maximizes information required to estimate a pose within an indoor space along the horizontal direction,” (pages 3-4, para [0063]).
The horizontally arranged lidar sensor scans the environment in the horizontal direction. The horizonal scan plane includes producing scan data at a fixed height which reads on scanning with respect to a first height. The spatial data includes LiDAR scan data. Scanning depth includes measuring a distance with the lidar sensor/ laser scanner.);
“…the 3D map generating method of the present invention includes: collecting spatial data and image data on a specific space by using a lidar sensor and a camera sensor each provided at a collecting device; estimating a movement trajectory of the lidar sensor by using the spatial data; and generating 3D structure data on the specific space based on Structure-From-Motion (SFM) techniques (or simply “SFM”), by using the image data and the movement trajectory as input data,” (page 1, para [0012]).
“…the present invention provides a three-dimensional (3D) map generating method, including: collecting spatial data and image data on a specific space by using a lidar sensor and a camera sensor, and estimating a movement trajectory of the lidar sensor; extracting feature points from the image data, and matching the feature points among images; and converting the movement trajectory into a continuous trajectory, optimizing the feature points and pose information of the image data based on the continuous trajectory, thereby generating a 3D feature map with respect to the specific space,” (page 2, para [0020]).
“The mapping device 120 generates a 3D feature point map (feature map) for a specific space by using the collecting device 110. In this case, the mapping device 120 may estimate a movement trajectory of the lidar sensor 112a by using the spatial data, and may generate 3D structure data on the specific space based on Structure-From-Motion (SFM), by using the image data and the movement trajectory as input data,” (page 4, para [0069]).).
Y. Lee '310 is not relied upon teaching but H. Lee '296 teaches perform object detection on the captured image (“the main controller 400 performs 3D simultaneous localization and mapping (SLAM) using the LeGO-LOAM algorithm, and performs object detection using the YOLO v5 algorithm,” (page 3; Fig. 6-7; Fig. 9-10).
“The proposed system (1) uses the YOLO v5 algorithm for object recognition, calculates the 3D location information of the object recognized through the depth camera, and “publishes” it through ROS. A total of 10189 images were extracted by combining the “backpack”, “hat”, and “umbrella” class images of the Google Open Images V6 dataset and the “backpack” and “umbrella” class images of the MS coco dataset,” (pages 4-5).
“To obtain the coordinates of objects detected through YOLO, depth information was used using a depth camera (RGBD camera). When detecting an object and drawing a bounding box in YOLO, the central part of the bounding box was designated as the position of the object, and the coordinates of the object were obtained using the depth value of the designated part and the angle of view of the camera,” (page 4).
“The proposed system draws a 3D map with 3D LiDAR and recognizes an object using a depth camera on a two-wheeled mobile robot,” (page 6).
After combination the image/ frame/ data captured by the depth/ RGBD camera becomes Y. Lee '310’s captured image. YOLO is operating on image frames.); and
generating a spatial map according to the performed object detection (“Simultaneous multi-object recognition is performed using a depth camera, and the recognized objects are displayed on the 3D map to complete a semantic 3D map,” (page 6; Fig 1).).
Before the effective filling date of the claimed invention, it would have been obvious to one having ordinary skill in the art to apply the teachings of H. Lee '296 to Y. Lee '310. The motivation would have been to improve the detail, information, and usefulness of a spatial map.
Y. Lee '310 in view of H. Lee '296 is not relied upon teaching but Ebrahimi Afrouzi teaches memory can store a program for generating a spatial map (“The functionality described herein may be provided by one or more processors of one or more computers executing code stored on a tangible, non-transitory, machine readable medium. In some cases, notwithstanding use of the singular term “medium,” the instructions may be distributed on different storage devices associated with different computing devices, for instance, with each computing device having a different subset of the instructions, an implementation consistent with usage of the singular term “medium” herein,” (page 173-174, para [0908]). The computer program comprises code / instructions.).
Before the effective filling date of the claimed invention, it would have been obvious to one having ordinary skill in the art to apply the teachings of Ebrahimi Afrouzi to Y. Lee '310 in view of H. Lee '296. The motivation would have been to enable automation of tasks and / or enable software / program execution. Additional motivation would have been to improve performance and efficiency.
Regarding claim 1, it is rejected using the same citations and rationales described in the rejection of claim 14.
Regarding claim 15, Y. Lee '310 in view of H. Lee '296 in further view of Ebrahimi Afrouzi teaches the electronic device of claim 14, wherein, in the generating the spatial map, the at least one processor is configured to: based on an object being detected from the captured image, obtain depth values at a plurality of spots along a second height different from the first height in the target area, based on the captured image (H. Lee '296; “It is also possible to calculate object coordinates with higher accuracy by using a LiDAR sensor to calculate the coordinates and depth values of objects detected by the camera,” (page 4).
“The proposed system (1) uses the YOLO v5 algorithm for object recognition, calculates the 3D location information of the object recognized through the depth camera…” (H. Lee '296; pages 4-5).
After combination the image/ frame/ data captured by the depth/ RGBD camera becomes Y. Lee '310’s captured image. YOLO is operating on image frames. 3D location information includes depth values.
Before the effective filling date of the claimed invention, it would have been obvious to one having ordinary skill in the art to apply the teachings of H. Lee '296 to Y. Lee '310 in view of Ebrahimi Afrouzi. The motivation would have been to improve the scale, accuracy, and usefulness of a spatial map.
Ebrahimi Afrouzi; “FIG. 1A illustrates a robot 6400 taking sensor readings 6401 using a sensor, such as a two-and-a-half dimensional LIDAR. The sensor may observe the environment in layers. For example, FIG. 1B illustrates an example of a first layer 6402 observed by the sensor at a height 10 cm above the driving surface, a second layer 6403 at a height 40 cm above the driving surface, a third layer 6404 at a height 80 cm above the driving surface, a fourth layer 6405 at a height 120 cm above the driving surface, and a fifth layer 6406 at a height 140 cm from the driving surface, corresponding with the five measurement lines in FIG. 1A,” (page 9, para [0249]; Fig 1A; Fig 1B).
“The distances to objects 2903 always fall along the same height in each of the captured images as a two-and-a-half dimensional LIDAR measured the distances,” (Ebrahimi Afrouzi; pages 52-53, para [0422]).
The depth values include distances measured by the two-and-a-half dimensional LIDAR. The plurality of spots includes multiple measured points/ distances along a measurement line. From Figure 1A and Figure 1B it is clear that measurements are obtained at a plurality of spots along the fourth layer 6405. A second height can include any of the multiple LiDAR layers at different heights (40 cm, 80 cm, 120 cm, 140 cm) in comparison to a first layer at a first height of 10 cm.
Before the effective filling date of the claimed invention, it would have been obvious to one having ordinary skill in the art to apply the teachings of Ebrahimi Afrouzi to Y. Lee '310 in view of H. Lee '296. The motivation would have been to improve the completeness, detail, and accuracy of a three dimensional / spatial map.),
and generate the spatial map, based on the obtained depth values (H. Lee '296; “The main control unit 400 is installed with a Robot Operating System (ROS), performs 3D SLAM (simultaneous localization and mapping) based on the collected information of the 3D lidar sensor 100 to generate a 3D map, and generates a 3D map, Based on the collected information of (200), multi-object recognition is performed and the recognized object is displayed on the 3D map,” (page 3).
“LiDAR (Light Detection and Ranging) is a sensor that uses light to measure distance and detect objects,” (H. Lee '296; page 2).
The collected information of the 3D LiDAR sensor includes distance measurements. Obtained depth values includes distance measurements. The spatial map includes H. Lee '296’s 3D map.).
Before the effective filling date of the claimed invention, it would have been obvious to one having ordinary skill in the art to apply the teachings of H. Lee '296 to Y. Lee '310 in view of Ebrahimi Afrouzi. The motivation would have been to improve the scale, accuracy, and usefulness of a spatial map.
Regarding claim 2, it is rejected using the same citations and rationales described in the rejection of claim 15.
Regarding claim 12, Y. Lee '310 in view of H. Lee '296 in further view of Ebrahimi Afrouzi teaches the method of claim 1, further comprising: based on an object being detected from the captured image (Ebrahimi Afrouzi; “the camera of the robot may capture still images and record videos and may be a depth camera. For example, a camera may be used to capture images or videos in a first time interval and may be used as a depth camera emitting structured light in a second time interval… the camera output may be provided to an image processor for use by a user and to a microcontroller of the camera for depth sensing, obstacle detection, presence detection, etc. In some embodiments, the camera output may be processed locally on the robot by a processor,” (pages 7-8, para [0245]).),
determining an area at which the detected object is located, by comparing a border of the spatial map determined based only on the LiDAR scan data with a border of the spatial map determined based on the LiDAR scan data and the captured image (Ebrahimi Afrouzi; “FIG. 50 illustrates an example of a LIDAR local map 4700 generated by an algorithm in simulation. The LIDAR local map 4700 follows a robot 4701, with the robot 4701 centered within the LIDAR local map 4700. The LIDAR local map 4700 is overlaid on the global map illustrated in FIGS. 49A-49C. Obstacles 4702, hidden obstacles 4703, and open areas (i.e., free space) 4704 are added into the LIDAR local map based on LIDAR scans... During online navigation, the LIDAR local map may be updated based on LIDAR scans collected in real-time. Areas already observed by the LIDAR remain in the local map even when the LIDAR is no longer observing the area in its field of view until the areas are pushed out of the LIDAR local map due to the size of the LIDAR local map. Offset between actual location of obstacles and locations in the LIDAR local map may correspond with the offset between the position of the ground truth robot 4706 and the estimated position of the robot 4701,” (page 32, para [0338]; Figs. 49A-49C; Fig. 50).
“In some embodiments, online navigation uses a real-time local map, such as the LIDAR local map, in conjunction with a global map of the environment for more intelligent path planning. In some cases, the global map may be used to plan a global movement path and while executing the global movement path, the processor may create a real-time local map using fresh LIDAR scans. In some embodiments, the processor may synchronize the local map with obstacle information from the global map to eliminate paths planned through obstacles…” (Ebrahimi Afrouzi; pages 32-33, para [0339]).
The spatial map includes the LiDAR local map and / or the global map. The LIDAR local map is generated from LiDAR scan data. Borders include obstacles and edges. The LIDAR local map includes a border of the spatial map determined based only on the LiDAR scan data (see Fig. 50). After combination, Ebrahimi Afrouzi’s global map becomes Y. Lee '310’s 3D map generated based on the LiDAR scan data and the captured image. After combination, the global map includes a border of the spatial map determined based on the based on the LiDAR scan data and the captured image.
Overlaying the LiDAR local map on the global map includes comparing borders of the LiDAR local map with borders of the global map. Additionally, synchronizing the LiDAR local map with the obstacle information from the global map includes comparing borders of the LiDAR local map with borders of the global map.
Adding obstacles into the LIDAR local map based on LIDAR scans and eliminating paths planned through obstacles includes determining an area at which the detected object is located.);
and displaying the detected object on the determined area on the spatial map (Ebrahimi Afrouzi; “Obstacles 4702, hidden obstacles 4703, and open areas (i.e., free space) 4704 are added into the LIDAR local map based on LIDAR scans,” (page 32, para [0338]; Figs. 49A-49C; Fig. 50).).
Before the effective filling date of the claimed invention, it would have been obvious to one having ordinary skill in the art to apply the teachings of Ebrahimi Afrouzi to Y. Lee '310 in view of H. Lee '296. The motivation would have been to improve map accuracy. Additional motivation would have been to improve map accuracy of detected objects including during navigation. Additional motivation would have been to improve path planning and obstacle avoidance. Further motivation would have been to improve localization and / or determining a more accurate position and / or orientation of a device and / or object relative to 3D / spatial map.
Regarding claim 13, Y. Lee '310 in view of H. Lee '296 in further view of Ebrahimi Afrouzi teaches a non-transitory computer-readable recording medium having recorded thereon a computer program, which, when executed by a computer, performs the method of claim 1 (Ebrahimi Afrouzi; “The functionality described herein may be provided by one or more processors of one or more computers executing code stored on a tangible, non-transitory, machine readable medium. In some cases, notwithstanding use of the singular term “medium,” the instructions may be distributed on different storage devices associated with different computing devices, for instance, with each computing device having a different subset of the instructions, an implementation consistent with usage of the singular term “medium” herein,” (page 173-174, para [0908]). The computer program comprises code / instructions.).
Before the effective filling date of the claimed invention, it would have been obvious to one having ordinary skill in the art to apply the teachings of Ebrahimi Afrouzi to Y. Lee '310 in view of H. Lee '296. The motivation would have been to enable automation of tasks and / or enable software / program execution. Additional motivation would have been to improve performance and efficiency.
Allowable Subject Matter
Claims 3 and 16 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims. Regarding claim 3 the prior art or combination thereof fails to disclose or make obvious “determining, as the second height, a height at which the detected object is not detected in the target area; and obtaining depth values at the plurality of spots along the second height from the depth image, based on a result of the depth calibration.”
Claims 4-11 and 17-20 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims. Claims 4-11 and 17-20 are in allowable form due to their dependency on claim 3 and 16.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Guizilini (US 20230230264 A1) - teaches the state of the art of depth calibration, like that of claims 3-4 and 16-17.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ERICA G THERKORN whose telephone number is (571)272-2939. The examiner can normally be reached Monday - Friday 9:00am - 5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Devona Faulk can be reached at 571-272-7515. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ERICA G THERKORN/ Examiner, Art Unit 2618
/DEVONA E FAULK/ Supervisory Patent Examiner, Art Unit 2618
Prosecution Insights