DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Specification
The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed.
The following title is suggested:
METHOD AND APPARATUS FOR DETECTING AVPS MARKER BY EXTRACTING A FEATURE MAP THROUGH A BACKBONE NETWORK, A CENTER POINT HEATMAP AND FEATURE MAPS TO GENERATE CANDIDATE INFORMATION COMPRISING BOUNDING BOX AND CENTER POINTS.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1, 2, 4, 5 and 10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Weitong (IDS Document titled “Lightweight 2D barcode positioning algorithm for CenterNet network) in view of CornerNet.
Re claim 1: Weitong discloses a method of detecting a coded marker, the method comprising:
receiving an image and preprocessing the image (e.g. a barcode image can be received and preprocessed, which is taught on page 2, the first column.);
extracting a feature map of the image through a backbone network (e.g. a heat map is sent through a machine learning model to extract feature maps, or feature information, from the heat map, which is taught on page 2, column 2.);
extracting a center point heatmap for a coded marker (e.g. a center point position of the barcode area is extracted when the barcode is read through a CenterNet network structure, which is taught in column 2, page 2. The Lightweight CenterNet network has the same output as the CenterNet network, which is taught in column 1 on page 3 in the Lightweight CenterNet network section.),
a first feature map for a width and height of a bounding box, a second feature map for offsets for adjusting a center point, and a third feature map for the coded marker from the feature map using a plurality of head networks (e.g. the invention discloses extracting from a heat map, a map of features associated with the width and height, a map of features associated with the offset for a center point and a feature of maps associated with center point position, which is taught on page 2, column 2. This invention still has the same output as the CenterNet algorithm, which is stated on page 3, column 1.);
generating candidate detection information comprising the bounding box and the corner points based on the center point heatmap, the first feature map, the second feature map, and the third feature map (e.g. the CenterNet output outputs detection information comprising a bounding box and center points of the objects. This is based on the output of the three maps of features utilized to output the detection information. The Lightweight CenterNet outputs the same information as the CenterNet algorithm, which is taught in page 3, column 1.).
However, Weitong fails to specifically teach the features of a third feature map for corner points of the coded marker from the feature map, and
outputting final detection information by performing a modified non-maximum suppression on the candidate detection information.
However, this is well known in the art as evidenced by CornerNet. Similar to the primary reference, CornerNet discloses determining corner points within an image (same field of endeavor or reasonably pertinent to the problem).
CornerNet discloses a third feature map for corner points of the coded marker from the feature map (e.g. a map of features associated with the corner points within an image is output from a network head. This is output with other feature maps of offsets, which is taught on page 4, in section 3.1 Overview. Applying this concept to the barcode marker within the primary reference performs the feature of the above limitation.), and
outputting final detection information by performing a modified non-maximum suppression on the candidate detection information (e.g. in order to generate the final bounding box from the three outputs including the heatmaps, the inventio applies a non-maximum suppression (NMS) to the heatmap, which is taught in the section 4.2 Testing Details on page 8.).
Therefore, in view of CornerNet, it would have been obvious to one of ordinary skill before the effective filing date of the claimed invention was made to have the feature of a third feature map for corner points of the coded marker from the feature map, and outputting final detection information by performing a modified non-maximum suppression on the candidate detection information, incorporated in the device of Weitong, in order to include a corner points determined in a feature map included for determining a bounding box and performing NMS to the final output, which can eliminate the need for anchor box designation and assisting the network to better localize corners (as stated in CornerNet Abstract).
Re claim 2: However, Weitong fails to specifically teach the features of the method of claim 1, wherein preprocessing the image comprises performing at least one of cropping, resizing, normalization, or standardization on the image.
However, this is well known in the art as evidenced by CornerNet. Similar to the primary reference, CornerNet discloses determining corner points within an image (same field of endeavor or reasonably pertinent to the problem).
CornerNet discloses wherein preprocessing the image comprises performing at least one of cropping, resizing, normalization, or standardization on the image (e.g. the invention proposes preprocessing of cropping or scaling data, which is taught on page 7 in section 4.1 Training details.).
Therefore, in view of CornerNet, it would have been obvious to one of ordinary skill before the effective filing date of the claimed invention was made to have the feature of wherein preprocessing the image comprises performing at least one of cropping, resizing, normalization, or standardization on the image, incorporated in the device of Weitong, in order to perform preprocessing operations to an image, which can reduce overfitting (as stated in CornerNet Section 4.1 on page 7).
Re claim 4: Weitong discloses the method of claim 1, wherein the plurality of head networks comprises
a center point heatmap head trained to output the center point heatmap of a class corresponding to the coded marker from the feature map (e.g. a map of the features is output from a network head in figure 5 reflecting the center point. The Lightweight CenterNet still outputs the same outputs as the CenterNet, which is taught on page 3, column 1 in the Lightweight CenterNet network section 2.1.),
a dimension head trained to output the first feature map for the width and height of the bounding box corresponding to the center point (e.g. a map of features is output by a head that is trained to output the width and height of the bounding box corresponding to the center point of the object, which is seen in figure 5 and explained in column 2, section 1 titled CenterNet target detection network on page 2.),
an offset head trained to output the second feature map for the offsets for adjusting the center point (e.g. a head is trained to output an offset for adjusting the center point, which is seen in figure 5 and discussed in column 2, section 1 titled CenterNet target detection network on page 2.).
However, Weitong fails to specifically teach the features of a corner point head trained to output the third feature map for a relative position of each of the corner points with reference to the center point.
However, this is well known in the art as evidenced by CornerNet. Similar to the primary reference, CornerNet discloses determining corner points within an image (same field of endeavor or reasonably pertinent to the problem).
CornerNet discloses a corner point head trained to output the third feature map for a relative position of each of the corner points with reference to the center point (e.g. a map of features associated with the corner points within an image is output from a network head. This is output with other feature maps of offsets, which is taught on page 4, in section 3.1 Overview. Figure 4 shows the prediction modules that are trained in order to output a feature map associated with relative positions of corner points that are trained against a reference point. Figure 4 is shown on page 5 and the training is illustrated in figure 5 and explained on pages 4 and 5.).
.
Therefore, in view of CornerNet, it would have been obvious to one of ordinary skill before the effective filing date of the claimed invention was made to have the feature of a corner point head trained to output the third feature map for a relative position of each of the corner points with reference to the center point, incorporated in the device of Weitong, in order to include a corner points determined in a feature map included for determining a bounding box, which can eliminate the need for anchor box designation and assisting the network to better localize corners (as stated in CornerNet Abstract).
Re claim 5: Weitong discloses the method of claim 1, wherein the plurality of head networks are trained through multi-task learning (e.g. with the tasks of training for the identification of the center point, offsets and width associated with a height information being output, this process is considered as multi-task learning, which is discussed on page 2, column 2.).
Re claim 10: Weitong discloses an apparatus comprising one or more processors and a memory operably connected to the one or more processors, wherein:
the memory stores instructions causing the one or more processors to perform operations in response to execution of instructions by the one or more processors (e.g. on page 5, the CPU and GPU are disclosed that are used to run the software to perform the features of the invention. The software is stored on a memory that is executed by a CPU.), and the operations comprise:
receiving an image and preprocessing the image (e.g. a barcode image can be received and preprocessed, which is taught on page 2, the first column.);
extracting a feature map of the image through a backbone network (e.g. a heat map is sent through a machine learning model to extract feature maps, or feature information, from the heat map, which is taught on page 2, column 2.);
extracting a center point heatmap for a coded marker, a first feature map for a width and height of a bounding box, a second feature map for offsets for adjusting a center point, and a third feature map for the coded marker from the feature map using a plurality of head networks (e.g. the invention discloses extracting from a heat map, a map of features associated with the width and height, a map of features associated with the offset for a center point and a feature of maps associated with center point position, which is taught on page 2, column 2. This invention still has the same output as the CenterNet algorithm, which is stated on page 3, column 1.);
generating candidate detection information comprising the bounding box and the corner points based on the center point heatmap, the first feature map, the second feature map, and the third feature map (e.g. the CenterNet output outputs detection information comprising a bounding box and center points of the objects. This is based on the output of the three maps of features utilized to output the detection information. The Lightweight CenterNet outputs the same information as the CenterNet algorithm, which is taught in page 3, column 1.).
However, Weitong fails to specifically teach the features of a third feature map for corner points of the coded marker from the feature map, and
outputting final detection information by performing modified non-maximum suppression on the candidate detection information.
However, this is well known in the art as evidenced by CornerNet. Similar to the primary reference, CornerNet discloses determining corner points within an image (same field of endeavor or reasonably pertinent to the problem).
CornerNet discloses a third feature map for corner points of the coded marker from the feature map (e.g. a map of features associated with the corner points within an image is output from a network head. This is output with other feature maps of offsets, which is taught on page 4, in section 3.1 Overview. Applying this concept to the barcode marker within the primary reference performs the feature of the above limitation.), and
outputting final detection information by performing a modified non-maximum suppression on the candidate detection information (e.g. in order to generate the final bounding box from the three outputs including the heatmaps, the inventio applies a non-maximum suppression (NMS) to the heatmap, which is taught in the section 4.2 Testing Details on page 8.).
Therefore, in view of CornerNet, it would have been obvious to one of ordinary skill before the effective filing date of the claimed invention was made to have the feature of a third feature map for corner points of the coded marker from the feature map, and outputting final detection information by performing modified non-maximum suppression on the candidate detection information, incorporated in the device of Weitong, in order to include a corner points determined in a feature map included for determining a bounding box and performing NMS to the final output, which can eliminate the need for anchor box designation and assisting the network to better localize corners (as stated in CornerNet Abstract).
Claim(s) 3 is/are rejected under 35 U.S.C. 103 as being unpatentable over Weitong, as modified by CornerNet, as applied to claim 1 above, and further in view of Sun (CN-115346169 (filing date:8/8/2022)).
Re claim 3: Weitong discloses the method of claim 1, wherein the backbone network is a convolutional neural network (CNN) using an inverted residual block structure and DLA (deep layer aggregation) and is trained to extract a feature map of an input image (e.g. the invention discloses using an hourglass network that works with a DLA in order to be trained to extract the feature maps from the input image, which is taught in the CenterNet target detection network section on page 2, column 2. In the lightweight network on page 3, a CSPDarkNet53-tiny is used as the backbone.).
However, Weitong fails to specifically teach the features of the backbone network is a convolutional neural network (CNN) using an inverted residual block structure.
However, this is well known in the art as evidenced by Sun. Similar to the primary reference, Sun discloses image processing of an image using the MobileNet2 algorithm (same field of endeavor or reasonably pertinent to the problem).
Sun discloses the backbone network is a convolutional neural network (CNN) using an inverted residual block structure (e.g. the invention discloses a MobileNetV2 used as a backbone with inverted residual block structure to process image data, which is taught in ¶ [83] and [84].).
Therefore, in view of Sun, it would have been obvious to one of ordinary skill before the effective filing date of the claimed invention was made to have the feature of the backbone network is a convolutional neural network (CNN) using an inverted residual block structure, incorporated in the device of Weitong, in order to include a backbone network using an inverted residual block structure, which reduces the parameter model size and speeds up calculation of the image processing (as stated in Sun ¶ [83]).
Claim(s) 6-8 is/are rejected under 35 U.S.C. 103 as being unpatentable over Weitong, as modified by CornerNet, as applied to claim 1 above, and further in view of CenterNet and Zhou (IDS document titled “Objects as Points”).
Re claim 6: Weitong discloses the method of claim 1, wherein generating the candidate detection information comprises:
extracting, from the first feature map, width and height values of the bounding box corresponding to the preset number of center points (e.g. the invention discloses extracting, from a feature of maps, the width and height of the bounding box based on the CenterNet operation output being the same, as mentioned on page 3, the first column. A certain size of the data evaluated correspond to a certain number of center points.);
extracting, from the second feature map, offset values corresponding to the preset number of center points (e.g. the offset values for the center points is extracted from a map of features output, which the output of this information is illustrated in figure 5 shown on page 5.);
extracting, from the third feature map, point coordinates corresponding to the preset number of center points (e.g. the invention discloses extracting, from a map of features, a center point position, which is taught on page 2 in the CenterNet section.); and
generating bounding box information using the center point coordinates, the width values, the height values, the offset values (e.g. the overall bounding box information is generated using the center point position or coordinates, the width and height values and the offset values associated with the center position. This is seen in figure 5 on page 5 and the same output is discussed on page 3 in the Lightweight CenterNet network section 2.1.),
However, Weitong fails to specifically teach the features of extracting, from the center point heatmap, coordinates and confidence scores for a preset number of center points in an order of highest confidence score among coordinates having a peak pixel value;
extracting, from the third feature map, corner point coordinates corresponding to the preset number of center points; and
generating bounding box information using the center point coordinates, the width values, the height values, the offset values, and a scaling factor, and generating corner point information using the center point coordinates, the corner point coordinates, and the scaling factor.
However, this is well known in the art as evidenced by CornerNet. Similar to the primary reference, CornerNet discloses determining corner points within an image (same field of endeavor or reasonably pertinent to the problem).
CornerNet discloses extracting, from the third feature map, corner point coordinates corresponding to the preset number of center points (e.g. a map of features associated with the corner points within an image is output from a network head. This is output with other feature maps of offsets, which is taught on page 4, in section 3.1 Overview.); and
generating bounding box information using the center point coordinates, the width values, the height values, the offset values, and a scaling factor (e.g. the primary reference discloses the center point coordinates, the width and height values and the offset values for generating the bounding boxes. The secondary reference identifies the scaling used to aid in reducing overfitting, which is taught in section 4.1 Training details on page 7.).
Therefore, in view of CornerNet, it would have been obvious to one of ordinary skill before the effective filing date of the claimed invention was made to have the feature of extracting, from the third feature map, corner point coordinates corresponding to the preset number of center points; and generating bounding box information using the center point coordinates, the width values, the height values, the offset values, and a scaling factor, incorporated in the device of Weitong, in order to include a corner points determined in a feature map included for determining a bounding box, which can eliminate the need for anchor box designation and assisting the network to better localize corners (as stated in CornerNet Abstract).
However, the combination above fails to specifically teach the features of generating corner point information using the center point coordinates, the corner point coordinates, and the scaling factor.
However, an aspect of this is well known in the art as evidenced by CenterNet. Similar to the primary reference, CenterNet discloses using the Center point to calculate bounding box information (same field of endeavor or reasonably pertinent to the problem).
CenterNet discloses generating corner point information using the center point coordinates, the corner point coordinates, and the scaling factor (e.g. the system is scale-aware of a central region to the bounding box. The center point keypoint is located along with the corner points. These are combined with the system being scale-aware of the central region that is within the bounding box. These variables are used to generate the corner information associated with a bounding box, which is taught in the sections 3.1-3.4 under Training on pages 3-5.).
Therefore, in view of CenterNet, it would have been obvious to one of ordinary skill before the effective filing date of the claimed invention was made to have the feature of generating corner point information using the center point coordinates, the corner point coordinates, and the scaling factor, incorporated in the device of Weitong, as modified by CornerNet, in order to detect corners, the center with a scale of the region to generate bonding box data, which can improve precision and recall (as stated in CenterNet Abstract).
However, the combination above fails to specifically teach the feature of extracting, from the center point heatmap, coordinates and confidence scores for a preset number of center points in an order of highest confidence score among coordinates having a peak pixel value.
However, this is well known in the art as evidenced by Zhou. Similar to the primary reference, Zhou discloses performing object detection (same field of endeavor or reasonably pertinent to the problem).
Zhou discloses extracting, from the center point heatmap, coordinates and confidence scores for a preset number of center points in an order of highest confidence score among coordinates having a peak pixel value (e.g. the location of a center point and confidence values for one score and eight connected neighbors are extracted. The keypoint with the highest confidence level is extracted, which is performed for the image of interest. This is disclosed in section 4, Objects as Points on page 3.).
Therefore, in view of Zhou, it would have been obvious to one of ordinary skill before the effective filing date of the claimed invention was made to have the feature of extracting, from the center point heatmap, coordinates and confidence scores for a preset number of center points in an order of highest confidence score among coordinates having a peak pixel value, incorporated in the device of Weitong, as modified by CornerNet and CenterNet, in order to extract coordinates and confidence scores based on certain criteria of center points, which can improve the speed and accuracy of the bounding box based detector (as stated in Zhou Abstract).
Re claim 7: However, Weitong fails to specifically teach the feature of the method of claim 6, wherein the coordinates having a peak pixel value are coordinates with a pixel value equal to or greater than pixel values of eight adjacent coordinates.
However, this is well known in the art as evidenced by Zhou. Similar to the primary reference, Zhou discloses performing object detection (same field of endeavor or reasonably pertinent to the problem).
Zhou discloses wherein the coordinates having a peak pixel value are coordinates with a pixel value equal to or greater than pixel values of eight adjacent coordinates (e.g. the peak in the heatmap correspond to the coordinate of the peak that is the highest out of the keypoint and its 8 connected neighbors. This is taught in section 4 on page 3.).
Therefore, in view of Zhou, it would have been obvious to one of ordinary skill before the effective filing date of the claimed invention was made to have the feature of wherein the coordinates having a peak pixel value are coordinates with a pixel value equal to or greater than pixel values of eight adjacent coordinates, incorporated in the device of Weitong, as modified by CornerNet and CenterNet, in order to extract coordinates and confidence scores based on certain criteria of center points, which can improve the speed and accuracy of the bounding box based detector (as stated in Zhou Abstract).
Re claim 8: However, Weitong fails to specifically teach the features of the method of claim 6, wherein the scaling factor is a ratio between a size of the image and a size of the feature map.
However, an aspect of this is well known in the art as evidenced by CenterNet. Similar to the primary reference, CenterNet discloses using the Center point to calculate bounding box information (same field of endeavor or reasonably pertinent to the problem).
CenterNet discloses wherein the scaling factor is a ratio between a size of the image and a size of the feature map (e.g. the invention discloses a scale-aware factor that takes into consideration the ratio between the size of the center area to the overall heatmap area. This is explained in section 3.2 Object Detection as Keypoint Triplets on pages 3 and 4 and illustrated in figure 3.).
Therefore, in view of CenterNet, it would have been obvious to one of ordinary skill before the effective filing date of the claimed invention was made to have the feature of wherein the scaling factor is a ratio between a size of the image and a size of the feature map, incorporated in the device of Weitong, as modified by CornerNet, in order to detect corners, the center with a scale of the region to generate bonding box data, which can improve precision and recall (as stated in CenterNet Abstract).
Allowable Subject Matter
Claim 9 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter: The features of claim 9 below were not disclosed by the searched and/or cited prior art.
Re claim 9: The method of claim 1, wherein outputting the final detection information comprises:
calculating each IoU (Intersection over Union) between bounding boxes included in the candidate detection information;
calculating, for bounding boxes with an IoU equal to or greater than a threshold, an average of center point coordinates, an average of width values, an average of height values, an average of confidence score values, and an average of corner point coordinates; and
outputting the averages as the final detection information.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Regina discloses calculating average confidence scores.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CHAD S DICKERSON whose telephone number is (571)270-1351. The examiner can normally be reached Monday-Friday 10AM-6PM EST..
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Abderrahim Merouan can be reached at 571-270-5254. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CHAD DICKERSON/ Primary Examiner, Art Unit 2683