Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to the applicant’s arguments
The previous rejection is withdrawn. A new search was conducted. A new reference was found that is relevant. A new rejection is made herein. Applicant’s arguments are now moot in view of the new rejection of the claims.
Claims 1 and 18 are amended to recite and the primary reference is silent but ALDEHARI teaches “....obtain depth information of a scene in a first direction based on a first image captured by a
first sensor of the movable platform and a first image portion extracted from a third image
captured by a third sensor of the movable platform,
obtain depth information of a scene in a second direction based on a second image captured
by a second sensor of the movable platform and a second image portion extracted from the third
image captured by the third sensor of the movable platform....;
(see page 5 and 6 where the drone has a first image sensor and a second image sensor and a RF device and the three sensors provide a dataset of the scene in 80 images and see page 6-7 where the data is input into a neural network and a virtual coordinate system showing the drone in x, y, and z directions for classification of the RF data, first image data and the second image data in 3d and an identification and fusion of the images including depth of the features and location of the features in the neural network)
.... and the first image portion corresponds to the first overlapping field of view and the second image portion corresponds to the second overlapping field of view, the first image portion being different from the second image portion” (See FIG. 10-11 and page 7 where the neural network can take the data from the first and the second image portion with the third sensor being RF analyzer data and provide that the first image is a helicopter and the second image is a drone and classify them in 3 dimensions and provide a confusion matrix to indicate which image is which when the images are overlapping and see page 6 where the images are concatenated or not) “.
It would have been obvious for one of ordinary skill in the art before the effective filing date of the present disclosure to combine the teachings of ALDEHARI with the disclosure of STATE GRID with a reasonable expectation of success since ALDEHARI teaches the method can include a first and a second camera that can take 80 images with a third sensor being an RF analyzer. This can help when the drone objects are occluded and the RF analyzer can provide additional data. The RF analyzer can provide a hint. The first sensor and the second sensor and the RF analyzer data can be provided to a neural network where the neural network can provide a classification of the images as a helicopter or a second drone and the 3d position of the images in the neural network relative to the drone coordinate system. This can provide additional information to the drone can detect different images for an improved classification and collision avoidance. See page 5-7.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-7 and 18 are rejected under 35 U.S.C. sec, 103 as being unpatentable as obvious in view of Chinese Patent Pub. No.: CN105892489B to State Grid Intelligent Technology filed in 2016 and is before the filing date of 11-5-21 and in view of Chinese Patent Pub. No.’: CN 111107303 A to Chunghwa Picture Tubes Ltd filing in 2018 and in view of NPL, Mohammed Aledhari et al, Sensor Fusion for Drone Detection, 2021 IEEE 93rd Vehicular Technology Conference (VTC2021-Spring) (Sensor Fusion for Drone Detection | IEEE Conference Publication | IEEE Xplore) (April 2021)(hereinafter “ALEDHARI”).
PNG
media_image1.png
596
834
media_image1.png
Greyscale
In regard to claim 1, and claim 18, State Grid discloses “1. A control apparatus for a movable platform,
comprising:
at least one processor; and
at least one memory including computer program code,
where the at least one memory and the computer program code are configured, with the at least one processor, to cause the apparatus to at least: (As shown in Figure 1-2, an autonomous obstacle avoidance UAV system based on multi-sensor fusion includes an environment and obstacle data analysis and processing module 2, an ultrasonic device 5, a propeller 6, a power module 7, and an obstacle avoidance decision module 9 And frame 10; Environment and obstacle data analysis and processing module 2 is installed on the front outer surface of described frame 10, and this module is specifically processor, is used for realizing the analysis and processing to environment and obstacle data, inertial sensor 1 and the forward direction millimeter-wave radar 3, and the inertial sensor 1 and the forward direction millimeter-wave radar 3 are located above the environment and obstacle data analysis and processing module 2, the inside of the frame 10 is equipped with an obstacle avoidance decision-making module 9, and the ultrasonic device 5 is fixed by)
obtain depth information of a scene in a first direction based on a first image captured by a first sensor of the movable platform (see claim 1 where the device is a drone that has a environmental perception system that includes a binocular machine vision system for stereo matching and provides a depth of the image)
and a third image by a third sensor of the movable platform, respectively; (see Fig. 1-2 where the drone has sensors including 1. A radar sensor 2 and an ultrasonic sensor and 3. first and second binocular cameras)
obtain depth information of a scene in a second direction based on a second image captured by a second sensor of the movable platform and the third image by the third sensor of the movable platform, respectively; and (see claim 6 where the drone can determine Avoidance decision-making module determines avoidance decision according to the output result of environment and barrier Data Analysis Services module, by flying Control system drive power plant module realizes that unmanned plane hides peripheral obstacle；
When environmental structure perception building, together by the binocular machine vision system buildup of unmanned plane surrounding, it is empty to constitute big vision Between range three-dimension measuring system, using in binocular vision system camera receive environment in body surface characteristic point, and Anaglyph is obtained by Stereo matching, determines depth image later, and carries out environmental structure perception building；
When constructing space connected region, Disorder Model is initially set up, obstacle is carried out using the structure of improved space Octree Spatial cuboids containing entire scene are divided into eight sub- cube grids by three directions, are organized into one eight by description Fork tree；
If contained scenery dough sheet number is greater than given threshold value in a certain sub-cube grid, make for the sub-cube further Subdivision until above-mentioned subdivision process dough sheet number contained by each leaf node of Octree is respectively less than given threshold value, and is tied It closes direction of advance millimetre-wave radar echo-signal and obtains fuselage at a distance from barrier；
Construct space connected region further include: carry out local paths planning using Artificial Potential Field Method, four carried by unmanned plane All binocular machine vision systems obtain a new three dimensional closure environment；Scanning is combined into multiresolution barrier map, in office Portion carries out the local paths planning in unmanned plane during flying region in order to avoid barrier collision；
Implement avoidance movement, avoidance decision-making module is delivered to the steering engine of unmanned plane during flying device servo mechanism by control signal, and Flight control instruction is generated according to inertial guidance data and flight path, unmanned plane receives flight control instruction, to control unmanned plane Avoidance is carried out to barrier.)
control movement of the movable platform in space based on the depth information of the scene in both the first direction and the second direction; (see claims 1-5 where the drone can fly in all directions based on the depth information)
wherein the first sensor, the second sensor and the third sensor are mounted on the movable platform at substantially a same level; (see Fig. 1-2 where the drone has sensors including 1. A radar sensor 2 and an ultrasonic sensor and 3. first and second binocular cameras)
the first sensor has a first overlapping field of view with the third sensor to form a first binocular system to observe the scene in the first direction of the movable platform; and (see Fig. 1-2 where the drone has sensors including 1. A radar sensor 2 and an ultrasonic sensor and 3. first and second binocular cameras that overlap the field of view)”.
PNG
media_image2.png
758
702
media_image2.png
Greyscale
Chunghaw teaches “the second sensor has a second overlapping field of view with the third sensor to form a second binocular system to observe the scene in the second direction of the movable platform; the first direction being different from the second direction.” (see claims 1-6 and the abstract where the two sensors can include a overlapping image and then the first image area and the second image area are subject to image splicing and form a composite image) (The processor 110 is coupled to a plurality of cameras (e.g., the cameras C1, C2, C3) and the memory 120. The memory 120 stores a plurality of instructions. The processor 110 executes these instructions to operate the traffic image system 10. The processor 110 receives the first area image, the second area image and the third area image, and determines whether the area images are suitable for film splicing (image splicing). If the determination is positive, the processor 110 splices the region images to generate a composite image. The composite image is, for example, a panoramic image covering the visual field ranges (including the first visual field FOV1, the second visual field FOV2, and the third visual field FOV3) of the imaging devices C1, C2, and C3. The driver grasps the surrounding situation of the vehicle 100 by viewing the composite image. If the determination is that stitching is not appropriate, the processor 110 controls a display device (not shown) to separately display at least one of the first region image, the second region image, and the third region image to avoid providing an inappropriate composite image.)”.
It would have been obvious for one of ordinary skill in the art before the effective filing date of the present disclosure to combine the teachings of CHUNGWA with the disclosure of STATE GRID with a reasonable expectation of success since CHUNGWA teaches that the first camera and the second camera of the drone shown in the xy and yz direction can form a depth image that is an overlapping screen area om between the image plane a and b. A composite image of the overlapping area can be formed and provided for a composite image. Further, the processor 110 may further calculate pixel positions (i.e., seam lines) of the first-field intersection point L1 in the first area image 310 and the second area image 320, respectively, and determine whether to perform image stitching between the first area image 310 and the second area image 320 according to whether the corresponding frames of the first area image 310 and the second area image 320 at the pixel positions match.
Claim 1 and claim 18 are amended to recite and the primary reference is silent but ALDEHARI teaches “....obtain depth information of a scene in a first direction based on a first image captured by a
first sensor of the movable platform and a first image portion extracted from a third image
captured by a third sensor of the movable platform,
obtain depth information of a scene in a second direction based on a second image captured
by a second sensor of the movable platform and a second image portion extracted from the third
image captured by the third sensor of the movable platform....;
(see page 5 and 6 where the drone has a first image sensor and a second image sensor and a RF device and the three sensors provide a dataset of the scene in 80 images and see page 6-7 where the data is input into a neural network and a virtual coordinate system showing the drone in x, y, and z directions for classification of the RF data, first image data and the second image data in 3d and an identification and fusion of the images including depth of the features and location of the features in the neural network)
.... and the first image portion corresponds to the first overlapping field of view and the second image portion corresponds to the second overlapping field of view, the first image portion being different from the second image portion” (See FIG. 10-11 and page 7 where the neural network can take the data from the first and the second image portion with the third sensor being RF analyzer data and provide that the first image is a helicopter and the second image is a drone and classify them in 3 dimensions and provide a confusion matrix to indicate which image is which when the images are overlapping and see page 6 where the images are concatenated or not) “.
It would have been obvious for one of ordinary skill in the art before the effective filing date of the present disclosure to combine the teachings of ALDEHARI with the disclosure of STATE GRID with a reasonable expectation of success since ALDEHARI teaches the method can include a first and a second camera that can take 80 images with a third sensor being an RF analyzer. This can help when the drone objects are occluded and the RF analyzer can provide additional data. The RF analyzer can provide a hint. The first sensor and the second sensor and the RF analyzer data can be provided to a neural network where the neural network can provide a classification of the images as a helicopter or a second drone and the 3d position of the images in the neural network relative to the drone coordinate system. This can provide additional information to the drone can detect different images for an improved classification and collision avoidance. See page 5-7.
PNG
media_image3.png
522
754
media_image3.png
Greyscale
State Grid discloses “...2. The control apparatus according to claim 1, wherein the movable platform further comprises a body and an arm, the body being connected to the arm; (see Figure 1 where the drone has a first and a second arm with the propellors on the arms)
the first sensor, the second sensor, and the third sensor being mounted on the body; (see radar sensor, camera sensor and ultrasonic sensor being mounted on the body and claims 1-3)
and the arm is configured to mount a power system of the movable platform, wherein at least a portion of the arm is located below a plane in which the first sensor, second sensor and third sensor are located”. (see Figure 1 where the drone has a first and a second arm with the propellors on the arms and are below the body)
It would have been obvious for one of ordinary skill in the art before the effective filing date of the present disclosure to combine the teachings of CHUNGWA with the disclosure of STATE GRID with a reasonable expectation of success since CHUNGWA teaches that the first camera and the second camera of the drone shown in the xy and yz direction can form a depth image that is an overlapping screen area om between the image plane a and b. A composite image of the overlapping area can be formed and provided for a composite image. Further, the processor 110 may further calculate pixel positions (i.e., seam lines) of the first-field intersection point L1 in the first area image 310 and the second area image 320, respectively, and determine whether to perform image stitching between the first area image 310 and the second area image 320 according to whether the corresponding frames of the first area image 310 and the second area image 320 at the pixel positions match.
State Grid discloses “...3. The control apparatus according to claim 2, wherein the first sensor, the second sensor, and the third sensor are respectively located on a side of the removable platform facing outside of the body of the removable platform”. (see Fig. 1-2 where the drone has sensors including 1. A radar sensor 2 and an ultrasonic sensor and 3. first and second binocular cameras)
It would have been obvious for one of ordinary skill in the art before the effective filing date of the present disclosure to combine the teachings of CHUNGWA with the disclosure of STATE GRID with a reasonable expectation of success since CHUNGWA teaches that the first camera and the second camera of the drone shown in the xy and yz direction can form a depth image that is an overlapping screen area om between the image plane a and b. A composite image of the overlapping area can be formed and provided for a composite image. Further, the processor 110 may further calculate pixel positions (i.e., seam lines) of the first-field intersection point L1 in the first area image 310 and the second area image 320, respectively, and determine whether to perform image stitching between the first area image 310 and the second area image 320 according to whether the corresponding frames of the first area image 310 and the second area image 320 at the pixel positions match.
State Grid discloses “...4. The control apparatus according to claim 2, wherein the movable platform further comprises a fourth sensor facing downwardly from the movable platform, and
the at least one memory and the computer program code are further configured, with the at least one processor, to cause the apparatus to at least obtain depth information of a scene below the movable platform based on an image captured by the fourth sensor of the movable platform. (As shown in Figure 1-2, an autonomous obstacle avoidance UAV system based on multi-sensor fusion includes an environment and obstacle data analysis and processing module 2, an ultrasonic device 5, a propeller 6, a power module 7, and an obstacle avoidance decision module 9 And frame 10; Environment and obstacle data analysis and processing module 2 is installed on the front outer surface of described frame 10, and this module is specifically processor, is used for realizing the analysis and processing to environment and obstacle data, inertial sensor 1 and the forward direction millimeter-wave radar 3, and the inertial sensor 1 and the forward direction millimeter-wave radar 3 are located above the environment and obstacle data analysis and processing module 2, the inside of the frame 10 is equipped with an obstacle avoidance decision-making module 9, and the ultrasonic device 5 is fixed by) (see claim 1 where the device is a drone that has a environmental perception system that includes a binocular machine vision system for stereo matching and provides a depth of the image) (see Fig. 1-2 where the drone has sensors including 1. A radar sensor 2 and an ultrasonic sensor and 3. first and second binocular cameras)
It would have been obvious for one of ordinary skill in the art before the effective filing date of the present disclosure to combine the teachings of CHUNGWA with the disclosure of STATE GRID with a reasonable expectation of success since CHUNGWA teaches that the first camera and the second camera of the drone shown in the xy and yz direction can form a depth image that is an overlapping screen area om between the image plane a and b. A composite image of the overlapping area can be formed and provided for a composite image. Further, the processor 110 may further calculate pixel positions (i.e., seam lines) of the first-field intersection point L1 in the first area image 310 and the second area image 320, respectively, and determine whether to perform image stitching between the first area image 310 and the second area image 320 according to whether the corresponding frames of the first area image 310 and the second area image 320 at the pixel positions match.
Chunghaw teaches 5. The control apparatus according to claim 4, wherein, with the fourth sensor as a vertex, a field of view of the fourth sensor in a direction along a head of the body to a tail of the body is less than or equal to a field of view of the fourth sensor in a direction along a side of the body. (see claims 1-6 and the abstract where the two sensors can include a overlapping image and then the first image area and the second image area are subject to image splicing and form a composite image) (The processor 110 is coupled to a plurality of cameras (e.g., the cameras C1, C2, C3) and the memory 120. The memory 120 stores a plurality of instructions. The processor 110 executes these instructions to operate the traffic image system 10. The processor 110 receives the first area image, the second area image and the third area image, and determines whether the area images are suitable for film splicing (image splicing). If the determination is positive, the processor 110 splices the region images to generate a composite image. The composite image is, for example, a panoramic image covering the visual field ranges (including the first visual field FOV1, the second visual field FOV2, and the third visual field FOV3) of the imaging devices C1, C2, and C3. The driver grasps the surrounding situation of the vehicle 100 by viewing the composite image. If the determination is that stitching is not appropriate, the processor 110 controls a display device (not shown) to separately display at least one of the first region image, the second region image, and the third region image to avoid providing an inappropriate composite image.)
It would have been obvious for one of ordinary skill in the art before the effective filing date of the present disclosure to combine the teachings of CHUNGWA with the disclosure of STATE GRID with a reasonable expectation of success since CHUNGWA teaches that the first camera and the second camera of the drone shown in the xy and yz direction can form a depth image that is an overlapping screen area om between the image plane a and b. A composite image of the overlapping area can be formed and provided for a composite image. Further, the processor 110 may further calculate pixel positions (i.e., seam lines) of the first-field intersection point L1 in the first area image 310 and the second area image 320, respectively, and determine whether to perform image stitching between the first area image 310 and the second area image 320 according to whether the corresponding frames of the first area image 310 and the second area image 320 at the pixel positions match.
State Grid discloses “...6. The control apparatus according to claim 4, wherein an upper boundary of the field of view of the fourth sensor along a height direction of the movable platform coincides or intersects with a lower surface of the arm”. (see claim 6 where the drone can determine Avoidance decision-making module determines avoidance decision according to the output result of environment and barrier Data Analysis Services module, by flying Control system drive power plant module realizes that unmanned plane hides peripheral obstacle；
When environmental structure perception building, together by the binocular machine vision system buildup of unmanned plane surrounding, it is empty to constitute big vision Between range three-dimension measuring system, using in binocular vision system camera receive environment in body surface characteristic point, and Anaglyph is obtained by Stereo matching, determines depth image later, and carries out environmental structure perception building；
When constructing space connected region, Disorder Model is initially set up, obstacle is carried out using the structure of improved space Octree Spatial cuboids containing entire scene are divided into eight sub- cube grids by three directions, are organized into one eight by description Fork tree；
If contained scenery dough sheet number is greater than given threshold value in a certain sub-cube grid, make for the sub-cube further Subdivision until above-mentioned subdivision process dough sheet number contained by each leaf node of Octree is respectively less than given threshold value, and is tied It closes direction of advance millimetre-wave radar echo-signal and obtains fuselage at a distance from barrier；
Construct space connected region further include: carry out local paths planning using Artificial Potential Field Method, four carried by unmanned plane All binocular machine vision systems obtain a new three dimensional closure environment；Scanning is combined into multiresolution barrier map, in office Portion carries out the local paths planning in unmanned plane during flying region in order to avoid barrier collision；
Implement avoidance movement, avoidance decision-making module is delivered to the steering engine of unmanned plane during flying device servo mechanism by control signal, and Flight control instruction is generated according to inertial guidance data and flight path, unmanned plane receives flight control instruction, to control unmanned plane Avoidance is carried out to barrier.) (see claims 1-6 and the abstract where the two sensors can include a overlapping image and then the first image area and the second image area are subject to image splicing and form a composite image) (The processor 110 is coupled to a plurality of cameras (e.g., the cameras C1, C2, C3) and the memory 120. The memory 120 stores a plurality of instructions. The processor 110 executes these instructions to operate the traffic image system 10. The processor 110 receives the first area image, the second area image and the third area image, and determines whether the area images are suitable for film splicing (image splicing). If the determination is positive, the processor 110 splices the region images to generate a composite image. The composite image is, for example, a panoramic image covering the visual field ranges (including the first visual field FOV1, the second visual field FOV2, and the third visual field FOV3) of the imaging devices C1, C2, and C3. The driver grasps the surrounding situation of the vehicle 100 by viewing the composite image. If the determination is that stitching is not appropriate, the processor 110 controls a display device (not shown) to separately display at least one of the first region image, the second region image, and the third region image to avoid providing an inappropriate composite image.)
State Grid discloses “...7. The control apparatus according to claim 4, wherein the movable platform comprises at least two fourth sensors, the at least two fourth sensors being disposed in a direction from a head to a tail of the body. (As shown in Figure 1-2, an autonomous obstacle avoidance UAV system based on multi-sensor fusion includes an environment and obstacle data analysis and processing module 2, an ultrasonic device 5, a propeller 6, a power module 7, and an obstacle avoidance decision module 9 And frame 10; Environment and obstacle data analysis and processing module 2 is installed on the front outer surface of described frame 10, and this module is specifically processor, is used for realizing the analysis and processing to environment and obstacle data, inertial sensor 1 and the forward direction millimeter-wave radar 3, and the inertial sensor 1 and the forward direction millimeter-wave radar 3 are located above the environment and obstacle data analysis and processing module 2, the inside of the frame 10 is equipped with an obstacle avoidance decision-making module 9, and the ultrasonic device 5 is fixed by)”.
Claim 8 is rejected under 35 U.S.C. sec, 103 as being unpatentable as obvious in view of Chinese Patent Pub. No.: CN105892489B to State Grid Intelligent Technology filed in 2016 and is before the filing date of 11-5-21 and in view of Chinese Patent Pub. No.’: CN 111107303 A to Chunghwa Picture Tubes Ltd filing in 2018 and in view of European Patent Pub. No.: EP3072817B1 to Routim that was filed in 2015 and Aldehari.
PNG
media_image4.png
564
1088
media_image4.png
Greyscale
Routin teaches “...8. The control apparatus according to claim 7, wherein the movable platform further comprises a lighting assembly, the lighting assembly facing downwardly from the movable platform or the lighting assembly being disposed between two of the at least two fourth sensors”. (see Fig. 4 where the drone can have two illumination devices from the two arms on the surface 102 via projectors 200, 200)
It would have been obvious for one of ordinary skill in the art before the effective filing date of the present disclosure to combine the teachings of ROUTIN with the disclosure of STATE GRID with a reasonable expectation of success since ROUTIN teaches that an aircraft can include two arms to support an illumination system to illuminate a wide-ranging area from the projected arms. See abstract.
Claims 9 and 19 are rejected under 35 U.S.C. sec, 103 as being unpatentable as obvious in view of Chinese Patent Pub. No.: CN105892489B to State Grid Intelligent Technology filed in 2016 and is before the filing date of 11-5-21 and in view of Chinese Patent Pub. No.’: CN 111107303 A to Chunghwa Picture Tubes Ltd filing in 2018 and Aldehari.
PNG
media_image5.png
526
756
media_image5.png
Greyscale
In regard to claim 9 and 19, State Grid discloses “....9. The control apparatus according to claim 1, wherein the movable platform comprises a body, the body including a head, a tail, a first side and a second side between the head and the tail, the first side and the second side being disposed opposite each other; (see Fig. 1 where the drone has a body and arms and a propellor and sensors on the body)
the first sensor is disposed at a corner position of the head and the first side, the second sensor is disposed at a corner position of the tail and the second side, and the third sensor is provided at a corner position of the head and the second side; (see specification As shown in Figure 1-2, an autonomous obstacle avoidance UAV system based on multi-sensor fusion includes an environment and obstacle data analysis and processing module 2, an ultrasonic device 5, a propeller 6, a power module 7, and an obstacle avoidance decision module 9 And frame 10; Environment and obstacle data analysis and processing module 2 is installed on the front outer surface of described frame 10, and this module is specifically processor, is used for realizing the analysis and processing to environment and obstacle data, inertial sensor 1 and the forward direction millimeter-wave radar 3, and the inertial sensor 1 and the forward direction millimeter-wave radar 3 are located above the environment and obstacle data analysis and processing module 2, the inside of the frame 10 is equipped with an obstacle avoidance decision-making module 9, and the ultrasonic device 5 is fixed by The frame is installed on the top of the frame 10, and a binocular machine vision system 4 is installed around the bracket of the ultrasonic device 5, and a power module 7 is installed on both sides of the frame 10 through a fixed rod, and a power module 7 is installed on the output end of the power module 7. Propeller 6, inertial sensor 1, forward direction millimeter-wave radar 3, binocular machine vision system 4 and output of ultrasonic device 5 are electrically connected to the input of environment and obstacle data analysis and processing module 2, environment and obstacle data analysis The output end of the processing module 2 is electrically connected to the input end of the obstacle avoidance decision module 9, the output end of the obstacle avoidance decision module 9 is connected to the input end of the power module 7 through the flight control system 11, and the ultrasonic device 5 is semicircular, And the anti-interference device is installed in the ultrasonic device 5, and the bottom of the frame 10 is equipped with a retractable landing gear 8.)
a width of the head is less than a length of each of the first side or the second side; and
the at least one memory and the computer program code are further configured, with the at least one processor, to cause the apparatus to at least:
obtain the depth information of the scene in the first direction based on images captured by the third sensor as a monocular system; wherein
a first manner of obtaining the depth information of the scene in the first direction is different from a second manner of the scene in the first direction based on the images, the first manner of obtaining the depth information of the scene in the first direction being based on the image captured by the third sensor, the second manner of obtaining the depth information of the scene in the first direction being based on the images captured by the first sensor and the third sensor respectively” (see claims 1-7 where the drone includes radar, and depth information ultrasonic sensor and camera and provides a environmental mapping and stereo matching with the depth sensor ) .
Claims 10-13 are rejected under 35 U.S.C. sec, 103 as being unpatentable as obvious in view of Chinese Patent Pub. No.: CN105892489B to State Grid Intelligent Technology filed in 2016 and is before the filing date of 11-5-21 and in view of Chinese Patent Pub. No.’: CN 111107303 A to Chunghwa Picture Tubes Ltd filing in 2018 and in view of U.S.. Patent Pub. No.: US 20140153916 A1 to Kintner that was filed in 2013 and Aldehari.
Kintner teaches “...10. The control apparatus according to claim 1, wherein field of views of the at least three sensors together form a field of view being 360°in a horizontal direction”. (see abstract and Fig. 1)”.
It would have been obvious for one of ordinary skill in the art before the effective filing date of the present disclosure to combine the teachings of KINTNER with the disclosure of STATE GRID with a reasonable expectation of success since KINTNER teaches that a drone can include arms to support a camera system on each of the arms. This can provide to capture images from a wide-ranging area from the projected arms in a 360 degree pattern. See abstract.
PNG
media_image6.png
496
612
media_image6.png
Greyscale
Kintner teaches “...11. The control apparatus according to claim 1, wherein the movable platform comprises a body, the body including a head, a tail, a first side and a second side between the head and the tail, the first side and the second side being disposed opposite each other; and
the first sensor, the second sensor, and the third sensor are each disposed at a corner position of the head and one of the first side or the second side, or of the tail and one of the first side or the second side”. (see paragraph 119-120 and Fig 10a where the drone has panoramic cameras each in the corner position of the tail or head of the drone)
It would have been obvious for one of ordinary skill in the art before the effective filing date of the present disclosure to combine the teachings of KINTNER with the disclosure of STATE GRID with a reasonable expectation of success since KINTNER teaches that a drone can include arms to support a camera system on each of the arms. This can provide to capture images from a wide-ranging area from the projected arms in a 360 degree pattern. See abstract.
Kintner teaches “..12. The control apparatus according to claim 1, wherein the movable platform comprises a body;
a main optical axis of the third sensor has a non-zero angle with respect to a first axis in a direction along a head of the body to a tail of the body; or
the main optical axis of the third sensor has a non-zero angle with respect to a second axis in a direction along a side-to-side of the body. (see paragraph 119-120 and Fig 10a where the drone has panoramic cameras each in the corner position of the tail or head of the drone)
It would have been obvious for one of ordinary skill in the art before the effective filing date of the present disclosure to combine the teachings of KINTNER with the disclosure of STATE GRID with a reasonable expectation of success since KINTNER teaches that a drone can include arms to support a camera system on each of the arms. This can provide to capture images from a wide-ranging area from the projected arms in a 360 degree pattern. See abstract.
Kintner teaches “..13. The control apparatus according to claim 1, wherein the movable platform comprises a body, the body being connected to an arm, the arm extending outwardly from the body, each of the first sensor, the second sensor, and the third sensor being provided at an end of the arm away from the body, respectively. (see paragraph 119-120 and Fig 10a where the drone has panoramic cameras each in the corner position of the tail or head of the drone)
It would have been obvious for one of ordinary skill in the art before the effective filing date of the present disclosure to combine the teachings of KINTNER with the disclosure of STATE GRID with a reasonable expectation of success since KINTNER teaches that a drone can include arms to support a camera system on each of the arms. This can provide to capture images from a wide-ranging area from the projected arms in a 360 degree pattern. See abstract.
Claims 14 to 17 are rejected under 35 U.S.C. sec, 103 as being unpatentable as obvious in view of Chinese Patent Pub. No.: CN105892489B to State Grid Intelligent Technology filed in 2016 and is before the filing date of 11-5-21 and in view of Kintner and Aldehari.
PNG
media_image1.png
596
834
media_image1.png
Greyscale
State Grid discloses “14. A control apparatus for a movable platform, comprising:
at least one processor; and
at least one memory including computer program code, where the at least one memory and the computer program code are configured, with the at least one processor, to cause the apparatus to at least: (As shown in Figure 1-2, an autonomous obstacle avoidance UAV system based on multi-sensor fusion includes an environment and obstacle data analysis and processing module 2, an ultrasonic device 5, a propeller 6, a power module 7, and an obstacle avoidance decision module 9 And frame 10; Environment and obstacle data analysis and processing module 2 is installed on the front outer surface of described frame 10, and this module is specifically processor, is used for realizing the analysis and processing to environment and obstacle data, inertial sensor 1 and the forward direction millimeter-wave radar 3, and the inertial sensor 1 and the forward direction millimeter-wave radar 3 are located above the environment and obstacle data analysis and processing module 2, the inside of the frame 10 is equipped with an obstacle avoidance decision-making module 9, and the ultrasonic device 5 is fixed by)
obtain depth information of a scene based on an image captured by a first sensor of the movable platform and a second image by a second sensor of the movable platform, respectively; and(see claim 1 where the device is a drone that has a environmental perception system that includes a binocular machine vision system for stereo matching and provides a depth of the image) (see Fig. 1-2 where the drone has sensors including 1. A radar sensor 2 and an ultrasonic sensor and 3. first and second binocular cameras)
control movement of the movable platform in space based on the depth information of the scene, and (see claim 6 where the drone can determine Avoidance decision-making module determines avoidance decision according to the output result of environment and barrier Data Analysis Services module, by flying Control system drive power plant module realizes that unmanned plane hides peripheral obstacle；
When environmental structure perception building, together by the binocular machine vision system buildup of unmanned plane surrounding, it is empty to constitute big vision Between range three-dimension measuring system, using in binocular vision system camera receive environment in body surface characteristic point, and Anaglyph is obtained by Stereo matching, determines depth image later, and carries out environmental structure perception building；
When constructing space connected region, Disorder Model is initially set up, obstacle is carried out using the structure of improved space Octree Spatial cuboids containing entire scene are divided into eight sub- cube grids by three directions, are organized into one eight by description Fork tree；
If contained scenery dough sheet number is greater than given threshold value in a certain sub-cube grid, make for the sub-cube further Subdivision until above-mentioned subdivision process dough sheet number contained by each leaf node of Octree is respectively less than given threshold value, and is tied It closes direction of advance millimetre-wave radar echo-signal and obtains fuselage at a distance from barrier；
Construct space connected region further include: carry out local paths planning using Artificial Potential Field Method, four carried by unmanned plane All binocular machine vision systems obtain a new three dimensional closure environment；Scanning is combined into multiresolution barrier map, in office Portion carries out the local paths planning in unmanned plane during flying region in order to avoid barrier collision；
Implement avoidance movement, avoidance decision-making module is delivered to the steering engine of unmanned plane during flying device servo mechanism by control signal, and Flight control instruction is generated according to inertial guidance data and flight path, unmanned plane receives flight control instruction, to control unmanned plane Avoidance is carried out to barrier.) (see claims 1-5 where the drone can fly in all directions based on the depth information)
PNG
media_image6.png
496
612
media_image6.png
Greyscale
Kintner teaches “...wherein the movable platform comprises a body and an arm, the arm extending outwardly from the body, the arm being configured to mount a power system for the movable platform;
the first sensor is oriented toward outside of the movable platform and the second sensor is oriented toward an underside of the movable platform; and
a portion of the arm is disposed between a lower boundary of a field of view of the first sensor along a height direction of the movable platform and an upper boundary of a field of view of the second sensor along the height direction of the movable platform”. (see paragraph 119-120 and Fig 10a where the drone has panoramic cameras each in the corner position of the tail or head of the drone)
It would have been obvious for one of ordinary skill in the art before the effective filing date of the present disclosure to combine the teachings of KINTNER with the disclosure of STATE GRID with a reasonable expectation of success since KINTNER teaches that a drone can include arms to support a camera system on each of the arms. This can provide to capture images from a wide-ranging area from the projected arms in a 360 degree pattern. See abstract.
PNG
media_image7.png
654
594
media_image7.png
Greyscale
Claim 14 is amended to recite and the primary reference is silent but ALDEHARI teaches “....the upper boundary of the field of view of the second sensor along the height direction coincides or intersects with a lower surface of the arm.; (see FIG. 4 where there are two camera and an RF sensor on the bottom section of the drone and that can capture the arms; and page 5 and 6 where the drone has a first image sensor and a second image sensor and a RF device and the three sensors provide a dataset of the scene in 80 images and see page 6-7 where the data is input into a neural network and a virtual coordinate system showing the drone in x, y, and z directions for classification of the RF data, first image data and the second image data in 3d and an identification and fusion of the images including depth of the features and location of the features in the neural network) (See FIG. 10-11 and page 7 where the neural network can take the data from the first and the second image portion with the third sensor being RF analyzer data and provide that the first image is a helicopter and the second image is a drone and classify them in 3 dimensions and provide a confusion matrix to indicate which image is which when the images are overlapping and see page 6 where the images are concatenated or not) “.
It would have been obvious for one of ordinary skill in the art before the effective filing date of the present disclosure to combine the teachings of ALDEHARI with the disclosure of STATE GRID with a reasonable expectation of success since ALDEHARI teaches the method can include a first and a second camera that can take 80 images with a third sensor being an RF analyzer. This can help when the drone objects are occluded and the RF analyzer can provide additional data. The RF analyzer can provide a hint. The first sensor and the second sensor and the RF analyzer data can be provided to a neural network where the neural network can provide a classification of the images as a helicopter or a second drone and the 3d position of the images in the neural network relative to the drone coordinate system. This can provide additional information to the drone can detect different images for an improved classification and collision avoidance. See page 5-7.
Kintner teaches ‘...15. The control apparatus according to claim 14, wherein the first sensor is disposed at a corner position between a head of the body and a side of the body, or between a tail of the body and the side of the body”. (see Fig. 10e and paragraph 119-120)
It would have been obvious for one of ordinary skill in the art before the effective filing date of the present disclosure to combine the teachings of KINTNER with the disclosure of STATE GRID with a reasonable expectation of success since KINTNER teaches that a drone can include arms to support a camera system on each of the arms. This can provide to capture images from a wide-ranging area from the projected arms in a 360 degree pattern. See abstract.
Kintner teaches “...16. The control apparatus according to claim 14, wherein a main optical axis of the first sensor has a non-zero angle with respect to a first axis of the first sensor in a direction along a head of the body to a tail of the body; or
the main optical axis of the first sensor has a non-zero angle with respect to a direction of the first sensor along a side-to-side of the body”. (see Fig. 10e and paragraph 119-120)
It would have been obvious for one of ordinary skill in the art before the effective filing date of the present disclosure to combine the teachings of KINTNER with the disclosure of STATE GRID with a reasonable expectation of success since KINTNER teaches that a drone can include arms to support a camera system on each of the arms. This can provide to capture images from a wide-ranging area from the projected arms in a 360 degree pattern. See abstract.
Kintner teaches “...17. The control apparatus according to claim 14, wherein
the movable platform comprises a body, the body being connected to an arm, the arm extending outwardly from the body, each of the first sensor, the second sensor, and the third sensor being provided at an end of the arm away from the body, respectively. (see Fig. 10e and paragraph 119-120)
It would have been obvious for one of ordinary skill in the art before the effective filing date of the present disclosure to combine the teachings of KINTNER with the disclosure of STATE GRID with a reasonable expectation of success since KINTNER teaches that a drone can include arms to support a camera system on each of the arms. This can provide to capture images from a wide-ranging area from the projected arms in a 360 degree pattern. See abstract.
Claims 20 is rejected under 35 U.S.C. sec, 103 as being unpatentable as obvious in view of Chinese Patent Pub. No.: CN105892489B to State Grid Intelligent Technology filed in 2016 and is before the filing date of 11-5-21 and in view of Chinese Patent Pub. No.’: CN 111107303 A to Chunghwa Picture Tubes Ltd filing in 2018
and in view of Korean Patent Pub. No.: KR101937669B1 that was filed in 2018 and Aldehari.
PNG
media_image8.png
496
376
media_image8.png
Greyscale
The 669 publication teaches “...20. The method according to claim 18, further comprising:
obtaining depth information of a scene above the movable platform based on an image captured by a binocular sensor mounted on the movable platform,
wherein the binocular sensor is configured to face upward of the movable platform when the movable platform moves”. (see abstract)
It would have been obvious for one of ordinary skill in the art before the effective filing date of the present disclosure to combine the teachings of the 669 publication with the disclosure of STATE GRID with a reasonable expectation of success since the 669 teaches that a drone can include a top side high definition camera to capture the field of view above the drone.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JEAN PAUL CASS whose telephone number is (571)270-1934. The examiner can normally be reached Monday to Friday 7 am to 7 pm; Saturday 10 am to 12 noon.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Scott A. Browne can be reached at 571-270-0151. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JEAN PAUL CASS/Primary Examiner, Art Unit 3666
Prosecution Insights