DETAILED ACTIONS
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Acknowledgment is made of applicant’s claim this application being in benefit of foreign priority from Taiwan Patent Application No. TW112137553 filed on September 28th, 2023.
Drawings
The 10-page drawings have been considered and placed on record in the file.
Status of Claims
Claims 1-10 are pending.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-6 and 9 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Chou et al., "Improving deep learning-based polyp detection using feature extraction and data augmentation" (2022), hereinafter referred to as Chou.
Claim 1
Chou discloses an object detection method (Chou, Fig. 5, Section 3, “The polyp image generated by the fusion module obtains the target object through the object detection algorithm.”), comprising:
receiving an image (Chou, Fig. 5, input); and
executing a deep neural network architecture (Chou, Fig. 5, Abstract, “This paper proposes a novel polyp detection method through deep learning, which uses a fusion mod ule combining feature extraction and data augmentation to enhance images. The Discrete Wavelet Transform (DWT) is applied to extract the texture features of polyps and strengthen the texture features that are not obvious in the polyp image. Then style-based GAN2 is used to enhance the image data, increase the image training data of YOLOv4, and let YOLOv4 learn more features of polyps.”) based on the image to obtain one or more object bounding boxes (Chou, Fig. 5, output, Section 3.3, “The Neck plays a very critical role in the target detection network. In polyp detection, the main goal of Neck is to generate multiple bounding boxes on the colonoscopy image”), wherein the deep neural network architecture includes a two-dimensional discrete wavelet transform (Abstract, “The Discrete Wavelet Transform (DWT) is applied to extract the texture features of polyps and strengthen the texture features that are not obvious in the polyp image. Then style-based GAN2 is used to enhance the image data, increase the image training data of YOLOv4, and let YOLOv4 learn more features of polyps.”, Section 3.3, “Our main aim is to improve or stabilize the accuracy of polyp detection. So far, we have completed the fusion module of feature extraction and data augmentation. We combined the previously completed module with the YOLOv4 network to obtain a deep learning polyp detection network”).
Claim 2
Chou discloses the object detection method according to claim 1 (Chou, Fig. 5, Section 3, “The polyp image generated by the fusion module obtains the target object through the object detection algorithm.”), wherein the deep neural network architecture (Chou, Fig. 5, Abstract, “This paper proposes a novel polyp detection method through deep learning, which uses a fusion mod ule combining feature extraction and data augmentation to enhance images. The Discrete Wavelet Transform (DWT) is applied to extract the texture features of polyps and strengthen the texture features that are not obvious in the polyp image. Then style-based GAN2 is used to enhance the image data, increase the image training data of YOLOv4, and let YOLOv4 learn more features of polyps.”) comprises:
a backbone network (Chou, Fig. 5, Backbone) , comprising the two-dimensional discrete wavelet transform (Abstract, “The Discrete Wavelet Transform (DWT) is applied to extract the texture features of polyps and strengthen the texture features that are not obvious in the polyp image., Section 3.3, “The Backbone is an updated version of Cross Stage partial Network Darknet53 (CSPDarknet53) using the YOLOv3 darknet53 frame work to extract the data to be recognized. The YOLOv4 backbone model for polyp detection belongs to CSPDarknet53 with a Spatial Pyramid Pooling (SPP) block to increase the learning ability of CNN and reduce the cost of memory.”);
a neck network, comprising a feature pyramid network, and configured to extract features from a transformation result of the two-dimensional discrete wavelet transform in the backbone network (Chou, Fig. 5, Neck network, Section 3.3, “The Neck plays a very critical role in the target detection network. In polyp detection, the main goal of Neck is to generate multiple bounding boxes on the colonoscopy image. The Neck structure of YOLOv4 mainly uses the methods of SPP and Path Aggregation Network (PANet) for feature integration”); and
a detection head, configured to obtain the one or more object bounding boxes of one or more objects from the neck network (Chou, Fig. 5, Head network, Section 3.3, “The head block in the structure of YOLOv4 architecture performs the final prediction. The prediction includes a bounding box to frame the polyp, a confidence score predicted to be a polyp, and a label to mark the polyp.”).
Claim 3
Chou discloses the object detection method according to claim 2 (Chou, Fig. 5, Section 3, “The polyp image generated by the fusion module obtains the target object through the object detection algorithm.”), wherein the detection head includes a large object detection head, a medium object detection head, and a small object detection head (Chou, Fig. 5, Head network includes a large medium and small detection head), each respectively configured to obtain the one or more object bounding boxes from multiple chunks of different sizes in the neck network (Chou, Section 3.3, “Considering the image resolution of the database and the best experimental results, we first scale the original image to a size of 416×416 in the YOLOv4algorithm. It divides the scaled image into a grid according to the size of the original image relative to the detection target. Assuming that the grid contains detection targets, the grid generates predicted bounding boxes and corresponding confidence scores. At the same time, for each predicted bounding box, a probability value is given to each class, and the class with the highest probability is the class judged as the bounding box.”).
Claim 4
Chou discloses the object detection method according to claim 2 (Chou, Fig. 5, Section 3, “The polyp image generated by the fusion module obtains the target object through the object detection algorithm.”), wherein the transformation result of the two-dimensional discrete wavelet transform comprises a sum result of at least two of three results obtained by filtering with a high-pass filter of the two-dimensional discrete wavelet transform (Chou, Section 3.1, “DWT has multiple resolutions to decompose the image into different frequency bands, has a good effect on the edge analysis of the image, and is often used to analyze the image texture. DWT is the same as wavelet transform in principle. The concept of image processing includes image pyramids and subband coding. These two concepts are essential for multi-resolution analysis. DWT can realize 2-dimension images through a two-channel filter bank [43], and includes a scaling function ϕLL (x,y) and three wavelets ψLH (x,y), ψHL(x,y), ψHH(x,y). Each scaling function or wavelet is the one-dimensional scaling function ϕ and corresponding wavelet ψ. When DWT decom poses the image, the original image is first analyzed by the low-pass filter in the horizontal direction and then down-sampled to get the low-frequency information of the image called L. After the high-pass filter analyzes the data in the horizontal direction, down-sampling is performed to get the high-frequency information of the image called H. Then, in the vertical direction of the image, L and H are analyzed by low-frequency filter and high frequency wave filter respectively and down-sampled.”, “combining the three different levels of features of DWT can achieve the effect of feature complementarity, so the sub-images used in this research are {HLi,HHi,LHi},i= 1,2,3.”, HH is high-frequency information filtered with high-pass filter and LH is low-frequency information filtered with high-pass filter).
Claim 5
Chou discloses the object detection method according to claim 4 (Chou, Fig. 5, Section 3, “The polyp image generated by the fusion module obtains the target object through the object detection algorithm.”), wherein the transformation result of the two-dimensional discrete wavelet transform comprises a concatenated result of the sum result and a result obtained without filtering by the high-pass filter of the two-dimensional discrete wavelet transform Chou, Section 3.1, “DWT has multiple resolutions to decompose the image into different frequency bands, has a good effect on the edge analysis of the image, and is often used to analyze the image texture. DWT is the same as wavelet transform in principle. The concept of image processing includes image pyramids and subband coding. These two concepts are essential for multi-resolution analysis. DWT can realize 2-dimension images through a two-channel filter bank [43], and includes a scaling function ϕLL (x,y) and three wavelets ψLH (x,y), ψHL(x,y), ψHH(x,y). Each scaling function or wavelet is the one-dimensional scaling function ϕ and corresponding wavelet ψ. When DWT decom poses the image, the original image is first analyzed by the low-pass filter in the horizontal direction and then down-sampled to get the low-frequency information of the image called L. After the high-pass filter analyzes the data in the horizontal direction, down-sampling is performed to get the high-frequency information of the image called H. Then, in the vertical direction of the image, L and H are analyzed by low-frequency filter and high frequency wave filter respectively and down-sampled.”, “combining the three different levels of features of DWT can achieve the effect of feature complementarity, so the sub-images used in this research are {HLi,HHi,LHi},i= 1,2,3.”, HL is high-frequency information filtered with low-pass filter).
Claim 6
Chou discloses the object detection method according to claim 2 (Chou, Fig. 5, Section 3, “The polyp image generated by the fusion module obtains the target object through the object detection algorithm.”), wherein the backbone network (Chou, Fig. 5, Backbone network) includes a convolutional neural network configured to obtain a convolution result (Chou, Section 3.3, “The input polyp image contains the color, texture, and shape of the polyp extracted through the convolutional backbone network”), and the neck network includes a feature extraction result obtained by concatenating the transformation result with the convolution result from the backbone network (Chou, Section 3.3, “Increasing the complexity of features and reducing the spatial resolution of the image will result in the loss of detailed information. In order to improve its shortcomings, we added a feature extraction method to decrease the missed detection rate of YOLOv4 for smaller objects.”, Abstract, “The Discrete Wavelet Transform (DWT) is applied to extract the texture features of polyps and strengthen the texture features that are not obvious in the polyp image. Then style-based GAN2 is used to enhance the image data, increase the image training data of YOLOv4, and let YOLOv4 learn more features of polyps.”).
Claim 9
Chou discloses a host for object detection (Chou, Section 4, “All the experiments in this study are implemented on Ubuntu 19.04 Linux computers”), comprising: one or more processors configured to execute multiple computer instructions stored in a non-volatile memory (Chou, Section 4, “The characteristics of the computer used to train and test the deep learning model are as follows: Intel Core i5-8500 3.0 GHz, 24 GB RAM, and NVIDIA GTX 2080 Ti GPU card. “) to implement the object detection method according to Claim 1 (Chou, Fig. 5).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 7 and 10 are rejected under 35 U.S.C. 103 as being unpatentable over Chou in view of Alaba et al., "WCNN3D:WaveletConvolutional Neural Network-Based 3D Object Detection for Autonomous Driving", (2022), hereinafter referred to as Alaba.
Claim 7
Chou discloses the performing the object detection method according to Claim 1 (Chou, Fig. 5, Section 3, “The polyp image generated by the fusion module obtains the target object through the object detection algorithm.”).
Chou does not explicitly disclose the method for estimating distances of object using images, comprising: using the one or more object bounding boxes and a corresponding parameter of the image to estimate a distances between one or more objects corresponding to the one or more object bounding boxes and a camera device that captured the image.
However, Alaba teaches the method for estimating distances of object using images (Alaba, Abstract, “Three-dimensional object detection is crucial for autonomous driving to understand the driving environment.”), comprising: using the one or more object bounding boxes and a corresponding parameter of the image (Alaba, Fig. 2, ” Architecture of the proposed WCNN3D object detection pedestrians and cyclists model for two levels of wavelet decomposition. Each convolutional block comprises three convolutional layers at each level of decomposition. We also used 1×1 convolutions to reduce the channel dimension of feature maps.”, “the detection head predicts classes of objects and regresses 3D bounding boxes for objects”) to estimate a distances between one or more objects corresponding to the one or more object bounding boxes and a camera device that captured the image (Alaba, Section 4.1, “The subscript c denotes the distance to the mean of all points in the pillar, whereas the subscript p denotes the offset from the pillar x and y centers. The LiDAR point at this stage is nine-dimensional (i.e., x, y, z, r, xc, yc, zc, xp, and yp). Most of the pillars are empty due to the sparsity of LiDAR points, so a dense tensor of size (D, P, N) was created to reduce the sparsity of the decoded LiDAR point, where D denotes the dimension, P the number of non-empty pillars per sample, and N is the number of points per pillar”, in the output point cloud in Fig . 2, the location of each bounding boxes determine the distance of each objects using the coordinates away from the ego-vehicle which has the camera device that captured the image).
Chou and Alaba are both considered to be analogous to the claimed invention because they are in the same field of convolutional neural network. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method as taught by Chou to incorporate the teachings of Alaba wherein the method for estimating distances of object using images, comprising: using the one or more object bounding boxes and a corresponding parameter of the image to estimate a distances between one or more objects corresponding to the one or more object bounding boxes and a camera device that captured the image. Such a modification is the result of combining prior art elements according to known methods to yield predictable results. The motivation for the proposed modification would have been to build a lightweight model without losing significant performance (Alaba, Abstract).
Claim 10
The combination of Chou in view of Alaba discloses a system (Chou, Section 4, “All the experiments in this study are implemented on Ubuntu 19.04 Linux computers”) for estimating the distances of object using images (Alaba, Abstract, “Three-dimensional object detection is crucial for autonomous driving to understand the driving environment.”), comprising: a host (Chou, Section 4, “All the experiments in this study are implemented on Ubuntu 19.04 Linux computers”) including one or more processors configured to execute multiple computer instructions stored in a non-volatile memory (Chou, Section 4, “The characteristics of the computer used to train and test the deep learning model are as follows: Intel Core i5-8500 3.0 GHz, 24 GB RAM, and NVIDIA GTX 2080 Ti GPU card. “) to implement the method for estimating distances of the object using images as described in Claim 7 (Alaba, Abstract, “Three-dimensional object detection is crucial for autonomous driving to understand the driving environment.”); and the camera device as described in Claim 7 (Alaba, Fig. 2, Lidar).
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Chou in view of Alaba in further view of Kang et al., "Camera and Radar Sensor Fusion for Robust Vehicle Localization via Vehicle Part Localization” (2020), hereinafter referred to as Kang.
Claim 8
The combination of Chou in view of Alaba discloses the method for estimating distances of object using images according to Claim 7 (Alaba, Fig. 2, ” Architecture of the proposed WCNN3D object detection pedestrians and cyclists model for two levels of wavelet decomposition. Each convolutional block comprises three convolutional layers at each level of decomposition. We also used 1×1 convolutions to reduce the channel dimension of feature maps.”, “the detection head predicts classes of objects and regresses 3D bounding boxes for objects”).
The combination of Chou in view of Alaba does not explicitly disclose wherein the corresponding parameter of the image includes a homography matrix, used to map one or more positions presented in the image to a ground.
However, Kang teaches wherein the corresponding parameter of the image includes a homography matrix, used to map one or more positions presented in the image to a ground (Kang, Abstract, “we propose a camera-radar sensor fusion framework for robust vehicle localization based on vehicle part (rear corner) detection and localization. The main idea of the proposed method is to reinforce the azimuth angle accuracy of the radar information by detecting and localizing the rear corner part of the target vehicle from an image. This part-based fusion approach enables accurate vehicle localization as well as robust performance with respect to occlusions. For efficient part detection, several candidate points are generated around the initial radar point”, Section 1, “A detected vehicle is represented as a bounding box in an image, and the contact point is defined as the point that overlaps both the bounding box and the road plane in the image. The distance is then estimated using the geometric relationship between this contact point and a camera installed on the vehicle. However, since a camera has low resolution in the longitudinal direction, a slight position error of the bounding box causes large longitudinal distance variation.”, Section II.C, “. Each initial radar point is converted into pixel coordinates in an image through the homography matrix, calibrated using the process described above. The candidate points are then generated around the pixel coordinates of the radar points with a uniform distribution. In each case, the mean of the uniform distribution is the pixel coordinate of the radar point and the size of the interval in the uniform distribution is dependent on pixel coordinate in the image plane”.
Chou, Alaba, and Kang are all considered to be analogous to the claimed invention because they are in the same field of object detection using deep learning methods. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method as taught by Chou to incorporate the teachings of Kang wherein the corresponding parameter of the image includes a homography matrix, used to map one or more positions presented in the image to a ground. Such a modification is the result of combining prior art elements according to known methods to yield predictable results. The motivation for the proposed modification would have been because Kang’s method provides significantly better localization performance in the lateral direction, with greatly reduced maximum errors (Kang, Abstract).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DENISE G ALFONSO whose telephone number is (571)272-1360. The examiner can normally be reached Monday - Friday 7:30 - 5:30.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amandeep Saini can be reached at (571)272-3382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DENISE G ALFONSO/Examiner, Art Unit 2662 /AMANDEEP SAINI/Supervisory Patent Examiner, Art Unit 2662
Prosecution Insights