DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Information Disclosure Statement
The information disclosure statement(s) (IDS) submitted on 01/02/2025, 01/29/2026, and 03/12/2026 have been considered and placed in the applicant file.
Claim Objections
Claims 1-2, 6-7, 11, 13-14 are objected to because of the following informalities:
Claim 1 contains bullet points at lines 3, 5, 7, and 10. Please remove each bullet point to correct for typographical and/or formatting issues in order to improve readability. Appropriate correction is required.
Claim 2 contains bullet points at lines 3, 5, 7. Please remove each bullet point to correct for typographical and/or formatting issues and improve readability. Appropriate correction is required.
Claim 6 contains bullet points at lines 5, 7, 11, 15 and 22. Please remove each bullet point to correct for typographical and/or formatting issues in order to improve readability. Appropriate correction is required.
In Claim 6, Line 1-4, the term “A method for configuring a classifier-localizer module for classifying and localizing objects in image sequences, said method comprising a step of initializing said classifier-localizer module and at least one iteration of the steps of:” should be changed to “A method for configuring a classifier-localizer module for classifying and localizing objects in image sequences, said method comprising a step of initializing said classifier-localizer module and at least one iteration of ” correct for typographical issues and/or improve readability. Appropriate correction is required.
Claim 7 contains bullet points at lines 3, 7, 11. Please remove each bullet point to correct for typographical and/or formatting issues and improve readability. Appropriate correction is required.
Claim 11 contains bullet points at lines 4 and 6. Please remove each bullet point to correct for typographical and/or formatting issues and improve readability. Appropriate correction is required.
Claim 13 contains bullet points at lines 3, 6, 8 and 11. Please remove each bullet point to correct for typographical and/or formatting issues and improve readability. Appropriate correction is required.
Claim 14 contains bullet points at lines 5, 7, 10 and 13. Please remove each bullet point to correct for typographical and/or formatting issues and improve readability. Appropriate correction is required.
In Claim 14, Line 1-4, the term “wherein said classifier-localizer module is configured by a configuring method comprising initializing said classifier-localizer module and at least one iteration of the steps of” should be changed to “wherein said classifier-localizer module is configured by a configuring method comprising initializing said classifier-localizer module and at least one iteration of ” correct for typographical issues and/or improve readability. Appropriate correction is required.
In Claim 18, Line 1-3, the term “A non-transitory computer readable medium having stored thereon instructions which, when executed by a processor, cause the processor to implement the steps of a method according to claim 1” should be changed to “A non-transitory computer readable medium having stored thereon instructions which, when executed by a processor, cause the processor to implement the ” correct for typographical issues and/or improve readability. Appropriate correction is required.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier.
Claims 1, 6 and 13-14 recite limitations that use words like “means” (or “step”) or similar terms with functional language and do invoke 35 U.S.C. 112(f):
Claim 1; recites the limitation, “determining by a classifier-localizer module…” [Line 5].
Claim 1; recites the limitation, “said classifier-localizer module being configured based on at least one reference image sequence to minimize a multi-objective loss function…” [Line 14-15].
Claim 6; recites the limitation, “determining, by said classifier-localizer module …” [Line 5].
Claim 6; recites the limitation, “reconfiguring said classifier-localizer module to minimize the multi-objective loss function…” [Line 22].
Claim 13; recites the limitation, “an obtaining module for obtaining…” [Line 3].
Claim 13; recites the limitation, “a classifier-localizer module for determining…” [Line 6].
Claim 13; recites the limitation, “said classifier-localizer module being configured based on at least one reference image sequence to minimize…” [Line 15].
Claim 14; recites the limitation, “determining, by said classifier-localizer module based on the at least one reference image sequence…” [Line 5].
Claim 14; recites the limitation, “reconfiguring said classifier-localizer module to minimize…” [Line 18].
Because these claim limitation(s) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, they are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
After a careful analysis, as disclosed above, and a careful review of the specification the following limitations in claims 1, 6 and 13-14:
“classifier-localizer module ” (Fig. 3, Paragraph [0015 and 0055-0057]-the classifier-localizer module is described as being configured to minimize a multi-objective loss function representative both of an objective of classification of one or more objects in an image sequence and of an objective of localization of these objects by learning classification and localization in parallel by the classifier-localizer module based on reference sequences during its configuration. The classifier-localizer module is further described as performing multiple iterations until convergence is reached. Fig. 3, illustrates a system with a black box labeled “X_NN” surrounding multiple other black boxes labeled “CNN”, “CLA NN” and “LOC_NN” (wherein the classifier-localizer module has sufficient structure associated with it, and it is understood as one or more neural networks)).
“obtaining module” (Fig. 1, Paragraph [0062]-the obtaining module is described as obtaining a sequence of one or more images captured by at least one camera. The classifier-localizer module is further described as performing multiple iterations until convergence is reached. Fig. 1, illustrates multiple black boxes, including a box labeled “CAM” connected to a box labeled “App”. (wherein the obtaining module has sufficient structure associated with it, and it is understood as a camera)).
If applicant does not intend to have these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4, 6-7, 11-14, 16, and 18-19 are rejected under 35 U.S.C. 103 as being unpatentable over HAO et al. (US 20210209368 A1), hereinafter referenced as HAO in view of BEIJBOM et al. (US 20220026917 A1), hereinafter referenced as BEIJBOM.
Regarding claim 1, HAO explicitly teaches a method for classifying and localizing at least one object in an image sequence (Fig. 7. Paragraph [0026]-HAO discloses FIG. 1 illustrates an example device 100 for performing object detection or recognition in video using a bidirectional pairing architecture. Device 100 includes any number of still image object detectors 101, 102, 103, 104, any number of similarity detectors 105, 106, 107, any number of concatenation modules 131, 132, 133, 134, and any number of election modules 151, 152, 153, 154. Further in paragraph [0057]-HAO discloses FIG. 6 illustrates an example bidirectional pairing architecture training network 600 for performing object detection or recognition in video. In paragraph [0070]-HAO discloses FIG. 7 is a flow chart illustrating a process 700 for training a bidirectional pairing architecture to perform object detection or recognition in video neural network to detect objects in video frames. Please also see Fig. 8-9 and 12), said method comprising:
obtaining a sequence of one or more images (Fig. 3, #301 called video data. Paragraph [0041]. Please also read paragraph [0058 and 0071]) captured by at least one camera (Fig. 3. Paragraph [0041]-HAO discloses network 300 receives video data 301 including a sequence of frames such as frames 111, 112, 113, 114 or any other frames discussed. Video data 301 may be a video stream of frames received from an imaging device (not shown). The frames of video data 301 may be reversed at reverse layer 302 such that processing is performed first in a first direction (e.g., forward) and subsequently in a second direction opposite the first direction. A pair of frames are established as pair of frames 303, 304 from video data 301 such that pair of frames 303, 304 are consecutive frames of video data 301 in either a temporal order or an order opposite the temporal order (e.g., pair of frames 303, 304 may be a current frame, i−1, and a subsequent frame, i, or a previous frame, i−1, and a current frame, i). Please also see Fig. 1 and 12, and read paragraph [0027, 0059, 0071 and 0125] (wherein device #100 and/or color camera #1222 may also be a camera)); and
determining, by a classifier-localizer module (Fig. 1, #300 called a network. Paragraph [0042 and 0057]) based on the obtained image sequence (Fig. 4. Paragraph [0042]-HAO discloses Network 300 includes a pair of still image object detectors including a still image object detector 305 and a paired still image object detector 306. Still image object detector 305 may receive frame i−1 and paired still image object detector 306 may receive frame i (in a forward processing mode) or still image object detector 305 may receive frame i and paired still image object detector 306 may receive frame i−1 (in a reverse processing mode). Still image object detectors 305, 306 may be any suitable still image object detectors such as region proposal network (faster R-CNN) detectors, region-based fully convolutional network (R-FCN) detectors, single-shot multibox (SSD) detectors, or you only look once (YOLO) detectors. Please also read paragraph [0057-0059 and 0070-0072]):
at least one class assigned to said at least one object, said at least one assigned class being selected from among a list of classes (Fig. 3. Paragraph [0035]-HAO discloses resultant object detection data 161, 162, 163, 164 may include any suitable data or data structures indicating resultant objects detected in respective ones of frames 114, 113, 112, 111. Resultant object detection data 161, 162, 163, 164 may indicate object detection localization (e.g., a bounding box index or bounding box location and size), a class (e.g., an index of one of multiple available object classes: person, building, car, dog, cattle, etc.), and a confidence scoring (e.g., a value between 0 and 1 indicating a confidence in the class) for one or more detected objects. Please also see Fig. 6-7, and read paragraph [000043, 0063 and 0076]); and
said classifier-localizer module being configured based on at least one reference image sequence (Fig. 6, #601 called video data. Paragraph [0058]-HAO discloses training network 300 receives video data 601 including multiple sequence of frames in analogy to frames 111, 112, 113, 114. Video data 601 may include multiple video streams of frames received as a training corpus to provide pretraining of system for object detection or recognition in video. Video data 601 may be training data including one or more objects within frames such that the objects are at least partially occluded. In paragraph [0065]-HAO discloses x.sup.p.sub.ij={1, 0} and y.sup.p.sub.ij={1, 0} may be indicators for matching the i-th default bounding box to the j-th ground truth bounding box of category p in the previous and reference frame. Given predicted boxes in the previous and reference frame of l.sub.f-1 and l.sub.f, and ground truth bounding boxes in the previous and reference frame of g.sub.f-1 and g.sub.f, and corresponding default bounding boxes in the previous and reference frame of d.sub.f-1 and d.sub.f, then the overall objective loss function L can be calculated by the multi-loss calculator 621 as a weighted sum of the localization loss (loc loss), the contrastive loss (contra loss) and the confidence loss (conf loss)) to minimize a multi-objective loss function (Fig. 3. Paragraph [0064]-HAO discloses the training objective of training network 300 may be to minimize the sum of a similarity loss between two object patches (contrastive loss), the localization loss for object bounding box detection (loc loss), and the confidence loss for object classification (conf loss). In paragraph [0077]-HAO discloses processing continues at operation 710, where a gradient is determined based on the multi-loss measurement and used to backward propagate the multi-loss measurement. The gradient may be calculated using partial derivatives with respect to each of the parameters of the multi-loss function. In some examples, the gradient based on the multi-loss measurement is propagated backwards through the entire training network or architecture, including the similarity detector, still image object detectors, and the election layer. In paragraph [0078]-HAO discloses processing continues at decision operation 711, where a determination is made as to whether a convergence is detected. If so, process 700 ends at end operation 712. If not, processing continues at operation 701 where another sequence of video frames for training is received and processing continues as discussed until convergence is met. The convergence decision may be based on the multi-loss measurement meeting a threshold, a maximum number of iterations being reached, etc.) representative both of an objective of classification and an objective of localization of objects in said at least one reference sequence (Fig. 3. Paragraph [0064]-HAO discloses training network 300 includes a localization (loc) and confidence (conf) loss calculator 618, a contrastive loss calculator 619, and a localization (loc) and confidence (conf) loss calculator 620. Training network 300 further includes a multi-loss calculator 621 coupled to localization and confidence loss calculator 618, contrastive loss calculator 619, and localization and confidence loss calculator 620. The training objective of training network 300 may be to minimize the sum of a similarity loss between two object patches (contrastive loss), the localization loss for object bounding box detection (loc loss), and the confidence loss for object classification (conf loss)).
Although HAO explicitly teaches at least one estimated position of said at least one object (Fig. 3. Paragraph [0035]-HAO discloses resultant object detection data 161, 162, 163, 164 may indicate object detection localization (e.g., a bounding box index or bounding box location and size)).
HAO fails to explicitly teach said at least one estimated position comprising at least one distance between an observer position and said at least one object.
However, BEIJBOM explicitly teaches at least one estimated position of said at least one object (Fig. 14. Paragraph [0113]-BEIJBOM discloses FIG. 13 is a block diagram of an image semantics network (ISN) 1300. The ISN 1300 takes input image 1301, predicts the class of each pixel in the image 1301 and outputs semantic segmentation data (e.g., a classification score) for each pixel in the image 1301 (wherein ISN 1300 is trained using an image dataset with annotated 2D bounding boxes and segmentation labels for classes). In paragraph [0116]-BEIJBOM discloses the SSD outputs are augmented to provide encoded 3D properties of the object along with a 2D bounding box and classification scores. The encoded 3D properties include information about the dimension, orientation and location of a 3D bounding box for the object in a format that can be decoded, along with the priorbox and camera intrinsic parameters, into a 3D bounding box in the vehicle coordinate system. Please also read paragraph [0085 and 0088]), said at least one estimated position comprising at least one distance between an observer position and said at least one object (Fig. 14. Paragraph [0118]-BEIJBOM discloses FIG. 14 illustrates an output of the ISN 1300 of FIG. 13. Multiple 3D bounding boxes are shown including bounding boxes 1401 (an SUV), 1402 (a pedestrian/jogger), 1403 (a motorcycle) and 1404 (a motorcycle). Bounding box 1404 is shown augmented with estimated geometry data for the object, including width (W), length (L) and height (H) and distance all in meters. Note that in FIG. 14 the estimated geometry data is compared to ground truth (GT) data (wherein loss functions include classification loss, localization loss and attribute loss). In paragraph [0120-0125]-BEIJBOM discloses for each priorbox 1500, designated as P (x.sub.p, y.sub.p, w.sub.p, h.sub.p), the detection head of ISN 1300 outputs a 12+K sized vector V of parameters: W, L, H correspond to object Width, Length and Height in meters. R is the radial distance from the camera center to the object center).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of HAO of having a method for classifying and localizing at least one object in an image sequence, said method comprising: obtaining a sequence of one or more images captured by at least one camera, with the teachings of BEIJBOM of having said at least one estimated position comprising at least one distance between an observer position and said at least one object.
Wherein HAO’s method having at least one estimated position of said at least one object said, at least one estimated position comprising at least one distance between an observer position and said at least one object.
The motivation behind the modification would have been to obtain a method that improves object detection and recognition, since both HAO and BEIJBOM concern image analysis and object detection/recognition. Wherein HAO’s provides improves improve object recognition and detection accuracy, while BEIJBOM’s systems and methods provide fast and accurate object detection. Please see HAO et al. (US 20210209368 A1), Paragraph [0020-0023] and BEIJBOM et al. (US 20220026917 A1), Abstract and Paragraph [0009].
Regarding claim 2, HAO in view of BEIJBOM explicitly teaches the method according to claim 1, HAO fails to explicitly teach wherein said determining comprises:
determining data encoded by an encoder based on the obtained image sequence;
determining said at least one class assigned by a classifier based on the encoded data; and
determining said at least one position estimated by a localizer based on the encoded data.
However, BEIJBOM explicitly teaches wherein said determining comprises:
determining data encoded by an encoder based on the obtained image sequence (Fig. 5. Paragraph [0113]-BEIJBOM discloses FIG. 13 is a block diagram of an image semantics network (ISN) 1300. ISN 1300 is a single stage, single image object detection network that is based on SSD. The ISN 1300 takes input image 1301, predicts the class of each pixel in the image 1301 and outputs semantic segmentation data (e.g., a classification score) for each pixel in the image 1301. The ISN 1300 is trained using a image dataset that includes images where each image is annotated with 2D bounding boxes and segmentation labels for classes in the image dataset. The ISN 1300 includes a backbone and detection head. The backbone is a fully convolutional neural network (FCNN), such as described in “PointPillars: Fast Encoders for Object Detection from Point Clouds, arXiv:1812.05784v2 [cs.LG] 7 May 2019”. In paragraph [0116]-BEIJBOM discloses the SSD outputs are augmented to provide encoded 3D properties of the object. Please also read paragraph [0090]);
determining said at least one class assigned by a classifier based on the encoded data (Fig. 5. Paragraph [0116]-BEIJBOM discloses the SSD outputs are augmented to provide encoded 3D properties of the object along with a 2D bounding box and classification scores. Please also read paragraph [0120-0125]); and
determining said at least one position estimated by a localizer based on the encoded data (Fig. 5. Paragraph [0120-0125]-BEIJBOM discloses for each priorbox 1500, designated as P (x.sub.p, y.sub.p, w.sub.p, h.sub.p), the detection head of ISN 1300 outputs a 12+K sized vector V of parameters: W, L, H correspond to object Width, Length and Height in meters. R is the radial distance from the camera center to the object center. ϕ.sub.x, ϕ.sub.y are combined to get the viewing angle ϕ. δ.sub.u, δ.sub.v, are combined with P to get the object 3D center projection on the image. c.sub.1, c.sub.2, . . . , c.sub.K are the classification (confidence) scores for K classes. In paragraph [0130]-BEIJBOM discloses given the priorbox 1501, the encoded 3D properties of the object (W, L, H, R, ϕ.sub.x, ϕ.sub.p, δ.sub.u, δ.sub.v), and camera intrinsics Q, the 3D bounding box 1502 for the object is determined).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of HAO in view of BEIJBOM of having a method for classifying and localizing at least one object in an image sequence, said method comprising: obtaining a sequence of one or more images captured by at least one camera, with the teachings of BEIJBOM of having wherein said determining comprises: determining data encoded by an encoder based on the obtained image sequence; determining said at least one class assigned by a classifier based on the encoded data; and determining said at least one position estimated by a localizer based on the encoded data.
Wherein HAO’s method having wherein said determining comprises: determining data encoded by an encoder based on the obtained image sequence; determining said at least one class assigned by a classifier based on the encoded data; and determining said at least one position estimated by a localizer based on the encoded data.
The motivation behind the modification would have been to obtain a method that improves object detection and recognition, since both HAO and BEIJBOM concern image analysis and object detection/recognition. Wherein HAO’s provides improves improve object recognition and detection accuracy, while BEIJBOM’s systems and methods provide fast and accurate object detection. Please see HAO et al. (US 20210209368 A1), Paragraph [0020-0023] and BEIJBOM et al. (US 20220026917 A1), Abstract and Paragraph [0009].
Regarding claim 3, HAO in view of BEIJBOM explicitly teaches the method according to claim 2, HAO fails to explicitly teach wherein said encoder implements one or more convolutions between the obtained image sequence and a number of filters.
However, BEIJBOM explicitly teaches wherein said encoder implements one or more convolutions between the obtained image sequence and a number of filters (Fig. 5. Paragraph [0114]-BEIJBOM discloses ISN 1300 includes two sub-networks. The top-down network can be characterized by a series of blocks. Each block operates at stride S (measured relative to the original input image 1301). A block has L 3×3 2D convolutional layers with F output channels, each followed by BatchNorm and a ReLU. The first convolution inside the layer has stride s/s.sub.in to ensure the block operates on stride S after receiving an input blob of stride S.sub.in. All subsequent convolutions in a block have stride 1. In paragraph [0116]-BEIJBOM discloses the SSD outputs are augmented to provide encoded 3D properties of the object along with a 2D bounding box and classification scores. Please also read paragraph [0113 and 0136]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of HAO in view of BEIJBOM of having a method for classifying and localizing at least one object in an image sequence, said method comprising: obtaining a sequence of one or more images captured by at least one camera, with the teachings of BEIJBOM of having wherein said determining comprises: determining data encoded by an encoder based on the obtained image sequence; determining said at least one class assigned by a classifier based on the encoded data; and determining said at least one position estimated by a localizer based on the encoded data.
Wherein HAO’s method having wherein said determining comprises: determining data encoded by an encoder based on the obtained image sequence; determining said at least one class assigned by a classifier based on the encoded data; and determining said at least one position estimated by a localizer based on the encoded data.
The motivation behind the modification would have been to obtain a method that improves object detection and recognition, since both HAO and BEIJBOM concern image analysis and object detection/recognition. Wherein HAO’s provides improves improve object recognition and detection accuracy, while BEIJBOM’s systems and methods provide fast and accurate object detection. Please see HAO et al. (US 20210209368 A1), Paragraph [0020-0023] and BEIJBOM et al. (US 20220026917 A1), Abstract and Paragraph [0009].
Regarding claim 4, HAO in view of BEIJBOM explicitly teaches the method according to claim 1, HAO further teaches wherein said classifier-localizer module (Fig. 1, #300 called a network. Paragraph [0042 and 0057]) comprises a neural network (Fig. 4. Paragraph [0042]-HAO discloses Network 300 includes a pair of still image object detectors including a still image object detector 305 and a paired still image object detector 306. Still image object detectors 305, 306 may be any suitable still image object detectors such as region proposal network (faster R-CNN) detectors, region-based fully convolutional network (R-FCN) detectors, single-shot multibox (SSD) detectors, or you only look once (YOLO) detectors).
Regarding claim 6, HAO explicitly teaches a method for configuring a classifier-localizer module (Fig. 1 and 3, #300 called a network. Paragraph [0042 and 0057]) for classifying and localizing objects in image sequences (Fig. 7. Paragraph [0043]-HAO discloses still image object detectors 305, 306 generate feature maps 307, 308, respectively. Feature map 307 may include one or two (or more) feature maps indicating localization information (loc) and confidence information or scores (conf) for the frame provided to still image object detector 305. Feature map 308 may include one or two (or more) feature maps indicating localization information (loc-p) and confidence information or scores (conf-p) for paired frame 304 provided to paired still image object detector 306), said method comprising initializing said classifier-localizer module and at least one iteration of the steps of (Fig. 6-7. Paragraph [0057]-HAO discloses FIG. 6 illustrates an example bidirectional pairing architecture training network 600 for performing object detection or recognition in video. Further in paragraph [0070]-HAO discloses FIG. 7 is a flow chart illustrating a process 700 for training a bidirectional pairing architecture to perform object detection or recognition in video neural network to detect objects in video frames. In paragraph [0078]-HAO discloses processing continues at decision operation 711, where a determination is made as to whether a convergence is detected. If so, process 700 ends at end operation 712. If not, processing continues at operation 701 where another sequence of video frames for training is received and processing continues as discussed until convergence is met. The convergence decision may be based on the multi-loss measurement meeting a threshold, a maximum number of iterations being reached, etc. Please also see Fig. 3, 8-9 and 12.):
determining, by said classifier-localizer module (Fig. 1 and 3, #300 called a network. Paragraph [0042 and 0057]) based on at least one reference image sequence (Fig. 6. Paragraph [0058]-HAO discloses training network 300 receives video data 601 including multiple sequence of frames in analogy to frames 111, 112, 113, 114. network 300 receives video data 301 including a sequence of frames such as frames 111, 112, 113, 114 or any other frames discussed herein. Video data 301 may be a video stream of frames received from an imaging device (not shown). The frames of video data 301 may be reversed at reverse layer 302 such that processing is performed first in a first direction (e.g., forward) and subsequently in a second direction opposite the first direction. As shown, a pair of frames are established as pair of frames 303, 304 from video data 301 such that pair of frames 303, 304 are consecutive frames of video data 301 in either a temporal order or an order opposite the temporal order (e.g., pair of frames 303, 304 may be a current frame, i−1, and a subsequent frame, i, or a previous frame, i−1, and a current frame, i). Please also see Fig. 3, and read paragraph [0043 and 0071]):
at least one class assigned to at least one object in said at least one reference sequence, said at least one assigned class being selected from among a list of classes (Fig. 3. Paragraph [0035]-HAO discloses resultant object detection data 161, 162, 163, 164 may include any suitable data or data structures indicating resultant objects detected in respective ones of frames 114, 113, 112, 111. Resultant object detection data 161, 162, 163, 164 may indicate object detection localization (e.g., a bounding box index or bounding box location and size), a class (e.g., an index of one of multiple available object classes: person, building, car, dog, cattle, etc.), and a confidence scoring (e.g., a value between 0 and 1 indicating a confidence in the class) for one or more detected objects. Please also see Fig. 6-7, and read paragraph [0063 and 0076]); and
reconfiguring said classifier-localizer module to minimize the multi-objective loss function (Fig. 7. Paragraph [0078]-HAO discloses processing continues at decision operation 711, where a determination is made as to whether a convergence is detected. If so, process 700 ends at end operation 712. If not, processing continues at operation 701 where another sequence of video frames for training is received and processing continues as discussed until convergence is met. The convergence decision may be based on the multi-loss measurement meeting a threshold, a maximum number of iterations being reached, etc. Please also see Fig. 3, 6 and 8, and read paragraph [0064-0065 and 0070-0077]).
Although HAO explicitly teaches at least one estimated position of said at least one object (Fig. 3. Paragraph [0035]-HAO discloses resultant object detection data 161, 162, 163, 164 may indicate object detection localization (e.g., a bounding box index or bounding box location and size));
evaluating a multi-objective loss function on the basis of said at least one assigned class, of at least one known class and of at least one known position of at least one object in said at least one reference sequence, the multi-objective loss function being representative both of an objective of classification and of an objective of localization of objects (Fig. 6. Paragraph [0064]-HAO discloses training network 300 includes a localization (loc) and confidence (conf) loss calculator 618, a contrastive loss calculator 619, and a localization (loc) and confidence (conf) loss calculator 620. Training network 300 further includes a multi-loss calculator 621 coupled to localization and confidence loss calculator 618, contrastive loss calculator 619, and localization and confidence loss calculator 620. The training objective of training network 300 may be to minimize the sum of a similarity loss between two object patches (contrastive loss), the localization loss for object bounding box detection (loc loss), and the confidence loss for object classification (conf loss). The overall objective loss function L can be calculated by the multi-loss calculator 621 as a weighted sum of the localization loss (loc loss), the contrastive loss (contra loss) and the confidence loss (conf loss). Please also see Fig. 3 and 7-8, and read paragraph [0065]);
HAO fails to explicitly teach said at least one estimated position comprising at least one distance between an observer position and said at least one object; evaluating a multi-objective loss function on the basis of said at least one estimated position.
However, BEIJBOM explicitly teaches said at least one estimated position comprising at least one distance between an observer position and said at least one object (Fig. 14. Paragraph [0118]-BEIJBOM discloses FIG. 14 illustrates an output of the ISN 1300 of FIG. 13. Multiple 3D bounding boxes are shown including bounding boxes 1401 (an SUV), 1402 (a pedestrian/jogger), 1403 (a motorcycle) and 1404 (a motorcycle). Bounding box 1404 is shown augmented with estimated geometry data for the object, including width (W), length (L) and height (H) and distance all in meters. Note that in FIG. 14 the estimated geometry data is compared to ground truth (GT) data. In paragraph [0120-0125]-BEIJBOM discloses for each priorbox 1500, designated as P (x.sub.p, y.sub.p, w.sub.p, h.sub.p), the detection head of ISN 1300 outputs a 12+K sized vector V of parameters: W, L, H correspond to object Width, Length and Height in meters. R is the radial distance from the camera center to the object center);
evaluating a multi-objective loss function on the basis of said at least one estimated position (Fig. 14. Paragraph [0113]-BEIJBOM discloses FIG. 13 is a block diagram of an image semantics network (ISN) 1300. ISN 1300 is a single stage, single image object detection network that is based on SSD. The ISN 1300 takes input image 1301, predicts the class of each pixel in the image 1301 and outputs semantic segmentation data (e.g., a classification score) for each pixel in the image 1301. The ISN 1300 is trained using a image dataset that includes images where each image is annotated with 2D bounding boxes and segmentation labels for classes in the image dataset. The ISN 1300 includes a backbone and detection head. The backbone is a fully convolutional neural network (FCNN), such as described in “PointPillars: Fast Encoders for Object Detection from Point Clouds, arXiv:1812.05784v2 [cs.LG] 7 May 2019”. Various loss functions can be employed ISN 1300 including but not limited to: classification loss, localization loss and attribute loss).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of HAO in view of BEIJBOM of having a method for classifying and localizing at least one object in an image sequence, said method comprising: obtaining a sequence of one or more images captured by at least one camera, with the teachings of BEIJBOM of having said at least one estimated position comprising at least one distance between an observer position and said at least one object; evaluating a multi-objective loss function on the basis of said at least one estimated position.
Wherein HAO’s method having said at least one estimated position comprising at least one distance between an observer position and said at least one object; evaluating a multi-objective loss function on the basis of said at least one assigned class, of said at least one estimated position, of at least one known class and of at least one known position of at least one object in said at least one reference sequence, the multi-objective loss function being representative both of an objective of classification and of an objective of localization of objects.
The motivation behind the modification would have been to obtain a method that improves object detection and recognition, since both HAO and BEIJBOM concern image analysis and object detection/recognition. Wherein HAO’s provides improves improve object recognition and detection accuracy, while BEIJBOM’s systems and methods provide fast and accurate object detection. Please see HAO et al. (US 20210209368 A1), Paragraph [0020-0023] and BEIJBOM et al. (US 20220026917 A1), Abstract and Paragraph [0009].
Regarding claim 7, HAO in view of BEIJBOM explicitly teaches the method according to claim 6, HAO further teaches wherein said evaluating the multi-objective loss function (Fig. 3. Paragraph [0064]-HAO discloses training network 300 includes a localization (loc) and confidence (conf) loss calculator 618, a contrastive loss calculator 619, and a localization (loc) and confidence (conf) loss calculator 620. Training network 300 further includes a multi-loss calculator 621 coupled to localization and confidence loss calculator 618, contrastive loss calculator 619, and localization and confidence loss calculator 620. Please also read paragraph [0077-0078]) comprises:
evaluating a classification loss function (Fig. 6. Paragraph [0064]-HAO discloses training network 300 includes a confidence (conf) loss calculator 618. Please also read paragraph [0065]) based on said at least one assigned class and on at least one known class associated with said at least one object of said at least one reference image sequence (Fig. 3. Paragraph [0043]-HAO discloses the confidence scores may be any number of confidence scores associated with a number of possible object types or classifications. For example, given the possible classifications of “cat,” “dog,” “car,” and “background,” the confidence information may be: (cat_score, dog_score, . . . , car_score, background_score), where cat_score, dog_score, car_score, and background_score are confidence scores for each of the possible classifications. Please also read paragraph [0042 and 0058]);
evaluating a localization loss function (Fig. 6. Paragraph [0064]-HAO discloses training network 300 includes a localization (loc) loss calculator 620. Please also read paragraph [0065]) based on said at least one estimated position and on at least one known position associated with said at least one object of said least one reference image sequence (Fig. 3. Paragraph [0043]-HAO discloses feature map 308 may include one or two (or more) feature maps indicating localization information (loc-p). The localization information may include predicted bounding box coordinates (x.sub.min, y.sub.min) and (x.sub.max, y.sub.max) or a predicted bounding box index (which may be used to access the coordinates and/or bounding box size). Please also read paragraph [0042, 0058 and 0068]);
and evaluating the multi-objective loss function based on the result of the classification loss function and on the result of the localization loss function (Fig. 3. Paragraph [0064]-HAO discloses training network 300 further includes a multi-loss calculator 621 coupled to localization and confidence loss calculator 618, contrastive loss calculator 619, and localization and confidence loss calculator 620. The training objective of training network 300 may be to minimize the sum of a similarity loss between two object patches (contrastive loss), the localization loss for object bounding box detection (loc loss), and the confidence loss for object classification (conf loss). Please also read paragraph [0065-0068]).
HAO fails to explicitly teach evaluating a localization loss function based on said at least one estimated position.
However, BEIJBOM explicitly teaches evaluating a localization loss function (Fig. 14. Paragraph [0113]-BEIJBOM discloses various loss functions can be employed ISN 1300 including but not limited to: classification loss, localization loss and attribute loss) based on said at least one estimated position (Fig. 14. Paragraph [0118]-BEIJBOM discloses FIG. 14 illustrates an output of the ISN 1300 of FIG. 13. Multiple 3D bounding boxes are shown including bounding boxes 1401 (an SUV), 1402 (a pedestrian/jogger), 1403 (a motorcycle) and 1404 (a motorcycle). Bounding box 1404 is shown augmented with estimated geometry data for the object, including width (W), length (L) and height (H) and distance all in meters. Note that in FIG. 14 the estimated geometry data is compared to ground truth (GT) data. In paragraph [0120-0125]-BEIJBOM discloses for each priorbox 1500, designated as P (x.sub.p, y.sub.p, w.sub.p, h.sub.p), the detection head of ISN 1300 outputs a 12+K sized vector V of parameters: W, L, H correspond to object Width, Length and Height in meters. R is the radial distance from the camera center to the object center).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of HAO in view of BEIJBOM of having a method configuring a classifier-localizer module, with the teachings of BEIJBOM of having evaluating a localization loss function based on said at least one estimated position.
Wherein HAO’s method having evaluating a localization loss function based on said at least one estimated position.
The motivation behind the modification would have been to obtain a method that improves object detection and recognition, since both HAO and BEIJBOM concern image analysis and object detection/recognition. Wherein HAO’s provides improves improve object recognition and detection accuracy, while BEIJBOM’s systems and methods provide fast and accurate object detection. Please see HAO et al. (US 20210209368 A1), Paragraph [0020-0023] and BEIJBOM et al. (US 20220026917 A1), Abstract and Paragraph [0009].
Regarding claim 11, HAO in view of BEIJBOM explicitly teaches the method according to claim 6, HAO further teaches wherein said at least one reference image sequence comprises at least one of the elements of the following group: one or more images captured by at least one camera; and one or more synthesized images (Fig. 2. Paragraph [0041]-HAO discloses network 300 receives video data 301 including a sequence of frames such as frames 111, 112, 113, 114 or any other frames discussed. Video data 301 may be a video stream of frames received from an imaging device (not shown). The frames of video data 301 may be reversed at reverse layer 302 such that processing is performed first in a first direction (e.g., forward) and subsequently in a second direction opposite the first direction. A pair of frames are established as pair of frames 303, 304 from video data 301 such that pair of frames 303, 304 are consecutive frames of video data 301 in either a temporal order or an order opposite the temporal order (e.g., pair of frames 303, 304 may be a current frame, i−1, and a subsequent frame, i, or a previous frame, i−1, and a current frame, i). Please also see Fig. 1 and 12, and read paragraph [0027, 0059, 0071 and 0125] (wherein device #100 and/or color camera #1222 may also be a camera)).
Regarding claim 12, HAO in view of BEIJBOM explicitly teaches the method according to claim 6, HAO further teaches comprising several iterations of said determining, evaluating the multi-objective loss function, and reconfiguring the classifier-localizer module (Fig. 7. Paragraph [0078]-HAO discloses processing continues at decision operation 711, where a determination is made as to whether a convergence is detected. If so, process 700 ends at end operation 712. If not, processing continues at operation 701 where another sequence of video frames for training is received and processing continues as discussed until convergence is met. The convergence decision may be based on the multi-loss measurement meeting a threshold, a maximum number of iterations being reached, etc. Please also see Fig. 3, 6 and 8, and read paragraph [0064-0065 and 0070-0077]).
Regarding claim 13, HAO explicitly teaches a device for classifying and localizing at least one object in an image sequence, said device comprising:
an obtaining module for obtaining a sequence of one or more images captured (Fig. 2, #301 called video data. Paragraph [0041]) by at least one camera (Fig. 2. Paragraph [0041]-HAO discloses network 300 receives video data 301 including a sequence of frames such as frames 111, 112, 113, 114 or any other frames discussed. Video data 301 may be a video stream of frames received from an imaging device (not shown). The frames of video data 301 may be reversed at reverse layer 302 such that processing is performed first in a first direction (e.g., forward) and subsequently in a second direction opposite the first direction. A pair of frames are established as pair of frames 303, 304 from video data 301 such that pair of frames 303, 304 are consecutive frames of video data 301 in either a temporal order or an order opposite the temporal order (e.g., pair of frames 303, 304 may be a current frame, i−1, and a subsequent frame, i, or a previous frame, i−1, and a current frame, i). Please also see Fig. 1 and 12, and read paragraph [0027, 0059, 0071 and 0125]); and
a classifier-localizer module (Fig. 3, #300 called a network. Paragraph [0042 and 0057]) for determining, based on the obtained image sequence (Fig. 3. Paragraph [0042]-HAO discloses Network 300 includes a pair of still image object detectors including a still image object detector 305 and a paired still image object detector 306. Still image object detector 305 may receive frame i−1 and paired still image object detector 306 may receive frame i (in a forward processing mode) or still image object detector 305 may receive frame i and paired still image object detector 306 may receive frame i−1 (in a reverse processing mode). Still image object detectors 305, 306 may be any suitable still image object detectors such as region proposal network (faster R-CNN) detectors, region-based fully convolutional network (R-FCN) detectors, single-shot multibox (SSD) detectors, or you only look once (YOLO) detectors. Please also read paragraph [0057-0059 and 0070-0072]):
at least one class assigned to said at least one object, said at least one assigned class being selected from among a list of classes (Fig. 3. Paragraph [0035]-HAO discloses resultant object detection data 161, 162, 163, 164 may include any suitable data or data structures indicating resultant objects detected in respective ones of frames 114, 113, 112, 111. Resultant object detection data 161, 162, 163, 164 may indicate object detection localization (e.g., a bounding box index or bounding box location and size), a class (e.g., an index of one of multiple available object classes: person, building, car, dog, cattle, etc.), and a confidence scoring (e.g., a value between 0 and 1 indicating a confidence in the class) for one or more detected objects. Please also see Fig. 6-7, and read paragraph [000043, 0063 and 0076]); and
said classifier-localizer module being configured based on at least one reference image sequence to minimize a multi-objective loss function representative both of an objective of classification and of an objective of localization of objects in said at least one reference image sequence (Fig. 6. Paragraph [0064]-HAO discloses training network 300 includes a localization (loc) and confidence (conf) loss calculator 618, a contrastive loss calculator 619, and a localization (loc) and confidence (conf) loss calculator 620. Training network 300 further includes a multi-loss calculator 621 coupled to localization and confidence loss calculator 618, contrastive loss calculator 619, and localization and confidence loss calculator 620. The training objective of training network 300 may be to minimize the sum of a similarity loss between two object patches (contrastive loss), the localization loss for object bounding box detection (loc loss), and the confidence loss for object classification (conf loss). The overall objective loss function L can be calculated by the multi-loss calculator 621 as a weighted sum of the localization loss (loc loss), the contrastive loss (contra loss) and the confidence loss (conf loss). Please also see Fig. 3 and 7-8, and read paragraph [0065 and 0070-0078]).
Although HAO explicitly teaches at least one estimated position of said at least one object, said at least one estimated position comprising at least one distance between an observer position and said at least one object;
HAO fails to explicitly teach said at least one estimated position comprising at least one distance between an observer position and said at least one object;
However, BEIJBOM explicitly teaches said at least one estimated position comprising at least one distance between an observer position and said at least one object (Fig. 14. Paragraph [0113]-BEIJBOM discloses FIG. 13 is a block diagram of an image semantics network (ISN) 1300. ISN 1300 is a single stage, single image object detection network that is based on SSD. The ISN 1300 takes input image 1301, predicts the class of each pixel in the image 1301 and outputs semantic segmentation data (e.g., a classification score) for each pixel in the image 1301. The ISN 1300 is trained using a image dataset that includes images where each image is annotated with 2D bounding boxes and segmentation labels for classes in the image dataset. The ISN 1300 includes a backbone and detection head. The backbone is a fully convolutional neural network (FCNN), such as described in “PointPillars: Fast Encoders for Object Detection from Point Clouds, arXiv:1812.05784v2 [cs.LG] 7 May 2019”. In paragraph [0116]-BEIJBOM discloses the SSD outputs are augmented to provide encoded 3D properties of the object. Please also read paragraph [0090]);.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of HAO of having a device for classifying and localizing at least one object in an image sequence, said device comprising: an obtaining module for obtaining a sequence of one or more images captured by at least one camera, with the teachings of BEIJBOM of having said at least one estimated position comprising at least one distance between an observer position and said at least one object.
Wherein HAO’s method having at least one estimated position of said at least one object, said at least one estimated position comprising at least one distance between an observer position and said at least one object;
The motivation behind the modification would have been to obtain a method that improves object detection and recognition, since both HAO and BEIJBOM concern image analysis and object detection/recognition. Wherein HAO’s provides improves improve object recognition and detection accuracy, while BEIJBOM’s systems and methods provide fast and accurate object detection. Please see HAO et al. (US 20210209368 A1), Paragraph [0020-0023] and BEIJBOM et al. (US 20220026917 A1), Abstract and Paragraph [0009].
Regarding claim 14, HAO in view of BEIJBOM explicitly teaches the device according to claim 13, HAO further teaches wherein said classifier-localizer module (Fig. 1 and 3, #300 called a network. Paragraph [0042 and 0057]) is configured by a configuring method comprising initializing said classifier-localizer module and at least one iteration of the steps of (Fig. 7. Paragraph [0070]-HAO discloses FIG. 7 is a flow chart illustrating a process 700 for training a bidirectional pairing architecture to perform object detection or recognition in video neural network to detect objects in video frames. In paragraph [0078]-HAO discloses processing continues at decision operation 711, where a determination is made as to whether a convergence is detected. If so, process 700 ends at end operation 712. If not, processing continues at operation 701 where another sequence of video frames for training is received and processing continues as discussed until convergence is met. The convergence decision may be based on the multi-loss measurement meeting a threshold, a maximum number of iterations being reached, etc. Please also see Fig. 3 and 6):
determining, by said classifier-localizer module based on the at least one reference image sequence (Fig. 6. Paragraph [0058]-HAO discloses training network 300 receives video data 601 including multiple sequence of frames in analogy to frames 111, 112, 113, 114. network 300 receives video data 301 including a sequence of frames such as frames 111, 112, 113, 114 or any other frames discussed herein. Video data 301 may be a video stream of frames received from an imaging device (not shown). The frames of video data 301 may be reversed at reverse layer 302 such that processing is performed first in a first direction (e.g., forward) and subsequently in a second direction opposite the first direction. As shown, a pair of frames are established as pair of frames 303, 304 from video data 301 such that pair of frames 303, 304 are consecutive frames of video data 301 in either a temporal order or an order opposite the temporal order (e.g., pair of frames 303, 304 may be a current frame, i−1, and a subsequent frame, i, or a previous frame, i−1, and a current frame, i). Please also see Fig. 3, and read paragraph [0043 and 0071]):
the at least one class assigned to at least one object in said at least one reference sequence, said at least one assigned class being selected from among a list of classes (Fig. 3. Paragraph [0035]-HAO discloses resultant object detection data 161, 162, 163, 164 may include any suitable data or data structures indicating resultant objects detected in respective ones of frames 114, 113, 112, 111. Resultant object detection data 161, 162, 163, 164 may indicate object detection localization (e.g., a bounding box index or bounding box location and size), a class (e.g., an index of one of multiple available object classes: person, building, car, dog, cattle, etc.), and a confidence scoring (e.g., a value between 0 and 1 indicating a confidence in the class) for one or more detected objects. Please also see Fig. 6-7, and read paragraph [000043, 0063 and 0076]); and
reconfiguring said classifier-localizer module to minimize the multi-objective loss function (Fig. 7. Paragraph [0070]-HAO discloses FIG. 7 is a flow chart illustrating a process 700 for training a bidirectional pairing architecture to perform object detection or recognition in video neural network to detect objects in video frames. In paragraph [0078]-HAO discloses processing continues at decision operation 711, where a determination is made as to whether a convergence is detected. If so, process 700 ends at end operation 712. If not, processing continues at operation 701 where another sequence of video frames for training is received and processing continues as discussed until convergence is met. The convergence decision may be based on the multi-loss measurement meeting a threshold, a maximum number of iterations being reached, etc. Please also see Fig. 3 and read paragraph [0071-0077]).
Although HAO explicitly teaches the at least one estimated position of said at least one object (Fig. 3. Paragraph [0035]-HAO discloses resultant object detection data 161, 162, 163, 164 may indicate object detection localization (e.g., a bounding box index or bounding box location and size));
evaluating the multi-objective loss function on the basis of said at least one assigned class, of at least one known class and of at least one known position of at least one object in said at least one reference sequence, the multi-objective loss function being representative both of an objective of classification and of an objective of localization of objects (Fig. 6. Paragraph [0064]-HAO discloses training network 300 includes a localization (loc) and confidence (conf) loss calculator 618, a contrastive loss calculator 619, and a localization (loc) and confidence (conf) loss calculator 620. Training network 300 further includes a multi-loss calculator 621 coupled to localization and confidence loss calculator 618, contrastive loss calculator 619, and localization and confidence loss calculator 620. The training objective of training network 300 may be to minimize the sum of a similarity loss between two object patches (contrastive loss), the localization loss for object bounding box detection (loc loss), and the confidence loss for object classification (conf loss). The overall objective loss function L can be calculated by the multi-loss calculator 621 as a weighted sum of the localization loss (loc loss), the contrastive loss (contra loss) and the confidence loss (conf loss). Please also see Fig. 3 and 7-8, and read paragraph [0065-0068 and 0070-0078]).
HAO fails to explicitly teach said at least one estimated position comprising at least one distance between an observer position and said at least one object; evaluating the multi-objective loss function on the basis of said at least one estimated position.
However, BEIJBOM explicitly teach said at least one estimated position comprising at least one distance between an observer position and said at least one object (Fig. 14. Paragraph [0118]-BEIJBOM discloses FIG. 14 illustrates an output of the ISN 1300 of FIG. 13. Multiple 3D bounding boxes are shown including bounding boxes 1401 (an SUV), 1402 (a pedestrian/jogger), 1403 (a motorcycle) and 1404 (a motorcycle). Bounding box 1404 is shown augmented with estimated geometry data for the object, including width (W), length (L) and height (H) and distance all in meters. Note that in FIG. 14 the estimated geometry data is compared to ground truth (GT) data. In paragraph [0120-0125]-BEIJBOM discloses for each priorbox 1500, designated as P (x.sub.p, y.sub.p, w.sub.p, h.sub.p), the detection head of ISN 1300 outputs a 12+K sized vector V of parameters: W, L, H correspond to object Width, Length and Height in meters. R is the radial distance from the camera center to the object center);
evaluating the multi-objective loss function on the basis of said at least one estimated position (Fig. 14. Paragraph [0113]-BEIJBOM discloses FIG. 13 is a block diagram of an image semantics network (ISN) 1300. ISN 1300 is a single stage, single image object detection network that is based on SSD. The ISN 1300 takes input image 1301, predicts the class of each pixel in the image 1301 and outputs semantic segmentation data (e.g., a classification score) for each pixel in the image 1301. The ISN 1300 is trained using a image dataset that includes images where each image is annotated with 2D bounding boxes and segmentation labels for classes in the image dataset. The ISN 1300 includes a backbone and detection head. The backbone is a fully convolutional neural network (FCNN), such as described in “PointPillars: Fast Encoders for Object Detection from Point Clouds, arXiv:1812.05784v2 [cs.LG] 7 May 2019”. Various loss functions can be employed ISN 1300 including but not limited to: classification loss, localization loss and attribute loss).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of HAO of having a device for classifying and localizing at least one object in an image sequence, said device comprising: an obtaining module for obtaining a sequence of one or more images captured by at least one camera, with the teachings of BEIJBOM of having said at least one estimated position comprising at least one distance between an observer position and said at least one object; evaluating the multi-objective loss function on the basis of said at least one estimated position.
Wherein HAO’s method having said at least one estimated position comprising at least one distance between an observer position and said at least one object; evaluating the multi-objective loss function on the basis of said at least one assigned class, of said at least one estimated position, of at least one known class and of at least one known position of at least one object in said at least one reference sequence, the multi-objective loss function being representative both of an objective of classification and of an objective of localization of objects.
The motivation behind the modification would have been to obtain a method that improves object detection and recognition, since both HAO and BEIJBOM concern image analysis and object detection/recognition. Wherein HAO’s provides improves improve object recognition and detection accuracy, while BEIJBOM’s systems and methods provide fast and accurate object detection. Please see HAO et al. (US 20210209368 A1), Paragraph [0020-0023] and BEIJBOM et al. (US 20220026917 A1), Abstract and Paragraph [0009].
Regarding claim 15, HAO in view of BEIJBOM explicitly teaches the system comprising a device according to claim 13, HAO further teaches and at least one camera configured to capture said one or more images of the sequence (Fig. 3. Paragraph [0043]-HAO discloses still image object detectors 305, 306 generate feature maps 307, 308, respectively. Feature map 307 may include one or two (or more) feature maps indicating localization information (loc) and confidence information or scores (conf) for the frame provided to still image object detector 305. Feature map 308 may include one or two (or more) feature maps indicating localization information (loc-p) and confidence information or scores (conf-p) for paired frame 304 provided to paired still image object detector 306. The localization information may include predicted bounding box coordinates (x.sub.min, y.sub.min) and (x.sub.max, y.sub.max) or a predicted bounding box index. The confidence scores may be any number of confidence scores associated with a number of possible object types or classifications).
Regarding claim 16, HAO in view of BEIJBOM explicitly teaches the system according to claim 15, HAO further teaches wherein said system is a surveillance system, or a navigation system (Fig. 3. Paragraph [0026]-HAO discloses [0026] FIG. 1 illustrates an example device 100 for performing object detection or recognition in video using a bidirectional pairing architecture (wherein device may be implemented in a motor vehicle platform). In paragraph [0043]-HAO discloses Device 1200 also may include navigation features 1212. Please also read paragraph [0116-0117]).
Regarding claim 18, HAO explicitly teaches a non-transitory computer readable medium having stored thereon instructions which, when executed by a processor, cause the processor to implement the steps of a method (Fig. 1 and 10-12. Paragraph [0018]-HAO discloses the material disclosed herein may also be implemented as instructions stored on a machine-readable medium, which may be read and executed by one or more processors. A machine-readable medium may include any medium and/or mechanism for storing or transmitting information in a form readable by a machine (e.g., a computing device). For example, a machine-readable medium may include read only memory (ROM); magnetic disk storage media; optical storage media).
HAO fails to explicitly teach a method according to claim 1.
However, HAO in view of BEIJBOM explicitly teaches a method according to claim 1 (please see the rejection for claim 15 further above).
Regarding claim 19, HAO explicitly teaches non-transitory computer readable medium having stored thereon instructions which, when executed by a processor, cause the processor to implement the steps of a method (Fig. 1 and 10-12. Paragraph [0018]-HAO discloses the material disclosed herein may also be implemented as instructions stored on a machine-readable medium, which may be read and executed by one or more processors. A machine-readable medium may include any medium and/or mechanism for storing or transmitting information in a form readable by a machine (e.g., a computing device). For example, a machine-readable medium may include read only memory (ROM); magnetic disk storage media; optical storage media).
However, HAO in view of BEIJBOM explicitly teaches a method according to claim 6 (please see the rejection for claim 6 further above).
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over HAO et al. (US 20210209368 A1), hereinafter referenced as HAO in view of BEIJBOM et al. (US 20220026917 A1), hereinafter referenced as BEIJBOM and in further view of KAMEN et al. (US 20220358648 A1), hereinafter referenced as KAMEN.
Regarding claim 5, HAO in view of BEIJBOM explicitly teaches the method according to claim 1, although HAO explicitly teaches wherein said one or more images of the obtained sequence are consecutive (Fig. 2. Paragraph [0027]-HAO discloses a sequence of video frames 111, 112, 113, 114 are provided in the following temporal order: F1, F2, F3, . . . FN. Video frames 111, 112, 113, 114 may include any suitable video frames, video pictures, sequence of video frames, group of pictures, groups of pictures, video data, or the like. In paragraph [0059]-HAO discloses the frames of video data 601 may be reversed at reverse layer 602 such that processing is performed first in a first direction (e.g., forward) and subsequently in a second direction opposite the first direction. As shown, a pair of frames are established as pair of frames 603, 604 from video data 601 such that pair of frames 603, 604 are consecutive frames of a sequence of video data 601 in either a temporal order or an order opposite the temporal order (e.g., pair of frames 603, 604 may be a current frame, i−1, and a subsequent frame, i, or a previous frame, i−1, and a current frame, i).Please also read paragraph [0028 and 0041]).
HAO in view of BEIJBOM fails to explicitly teaches and wherein the obtaining the image sequence comprises cropping said one or more captured images, one of said objects being centered on a determined image of the sequence.
However, KAMEN explicitly teaches and wherein the obtaining the image sequence (Fig. 1. Paragraph [0025]-KAMEN discloses FIG. 1 shows a method 100 for localizing and classifying lesions (or any other abnormality) in a medical image. In paragraph [0033]-KAMEN discloses FIG. 2 shows a network architecture of a multi-task network 200. In paragraph [0038]-KAMEN discloses for both single modality localization loss 212 and classification loss 214, a binary cross entropy loss function is chosen as the objective function. Combining the loss functions linearly with a weighting factor results in the overall loss that is optimized during the joint training procedure. Localization network 204 is trained with ground truth data comprising training medical images and corresponding lesion contours or approximate localization maps annotated at the center of lesions with a Gaussian blur applied around the center of the lesion. Classification network 208 is trained with ground truth data comprising training medical images, annotated with a binary label indicating the malignancy of a lesion, and corresponding localization maps. Please also read paragraph [0049 and 0074] (wherein training images may be synthesized)) comprises cropping said one or more captured images, one of said objects being centered on a determined image of the sequence (Fig. 2. Paragraph [0052]-KAMEN discloses a prostate segmentation algorithm may be leveraged to center the images around the prostate (or any other object of interest). Thus, by removing geometric variability, each of the plurality of images of mpMRI image 302 will have the same size, orientation, spacing, and position properties. In paragraph [0058]-KAMEN discloses during a training or offline stage, patches are cropped around clinically significant lesions (e.g., prostate lesions having a PI-RADS score greater than or equal to 3) from mpMRI images as training patches. The training patches are used to train the deep neural network for multi-class classificatio.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of HAO in view of BEIJBOM of having a method for classifying and localizing at least one object in an image sequence, said method comprising: obtaining a sequence of one or more images captured by at least one camera, with the teachings of KAMEN of having and wherein the obtaining the image sequence comprises cropping said one or more captured images, one of said objects being centered on a determined image of the sequence.
Wherein HAO’s method having and wherein the obtaining the image sequence comprises cropping said one or more captured images, one of said objects being centered on a determined image of the sequence.
The motivation behind the modification would have been to obtain a method that improves object detection and recognition, since both HAO and BEIJBOM concern image analysis and object detection/recognition. Wherein HAO’s provides improves improve object recognition and detection accuracy, while KAMEN’s systems and improve object recognition and training using machine learning. Please see HAO et al. (US 20210209368 A1), Paragraph [0020-0023] and KAMEN et al. (US 20220358648 A1), Abstract and Paragraph [0052 and 0074].
Claim 10 are rejected under 35 U.S.C. 103 as being unpatentable over HAO et al. (US 20210209368 A1), hereinafter referenced as HAO in view of BEIJBOM et al. (US 20220026917 A1), hereinafter referenced as BEIJBOM and in further view of LIU et al. (US 20190205643 A1), hereinafter referenced as LIU.
Regarding claim 10, HAO in view of BEIJBOM explicitly teaches the method according to claim 6, HAO in view of BEIJBOM fails to explicitly teach wherein said reconfiguring is performed using a gradient descent algorithm based on the result of the evaluation of the multi-objective loss function.
However, LIU explicitly teaches wherein said reconfiguring is performed using a gradient descent algorithm based on the result of the evaluation of the multi-objective loss function (Fig. 1. Paragraph [0044]-LIU discloses the overall objective function loss function of multitask deep neural network 304 is a weighted sum of localization loss, classification loss and attribute prediction loss. In the training stage, the weights of the entire network 304 can be optimized to minimize the overall objective function. Backpropagation, an abbreviation for “backward propagation of errors,” can be used in conjunction with the stochastic gradient descent method to train multitask deep neural network 304).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of HAO in view of BEIJBOM of having a method for configuring a classifier-localizer module, with the teachings of LIU of having wherein said reconfiguring is performed using a gradient descent algorithm based on the result of the evaluation of the multi-objective loss function.
Wherein HAO’s method having wherein said reconfiguring is performed using a gradient descent algorithm based on the result of the evaluation of the multi-objective loss function.
The motivation behind the modification would have been to obtain a method that improves object detection and recognition, since both HAO and LIU concern image analysis and object detection/recognition. Wherein HAO’s provides improves object recognition and detection accuracy, while LIU’s systems and improves localization and classification in machine learning. Please see HAO et al. (US 20210209368 A1), Paragraph [0020-0023] and LIU et al. (US 20190205643 A1), Abstract and Paragraph [0017 and 0032].
Claim 17 are rejected under 35 U.S.C. 103 as being unpatentable over HAO et al. (US 20210209368 A1), hereinafter referenced as HAO in view of BEIJBOM et al. (US 20220026917 A1), hereinafter referenced as BEIJBOM and in further view of CHOI et al. (US 20220051017 A1), hereinafter referenced as CHOI.
Regarding claim 17, HAO in view of BEIJBOM explicitly teach a system according to claim 15 (please see the rejection for claim 15 further above).
HAO in view of BEIJBOM fail to explicitly teaches an aircraft comprising a system.
However, CHOI explicitly teaches an aircraft comprising a system (Fig. 9A. Paragraph [0143]-CHOI discloses FIG. 9A illustrates an example of an autonomous vehicle 900.vehicle 900 may be an airplane (wherein FIG. 1 illustrates a diagram 100 of a deep learning object detection network using a localization and classification loss function). Please also see Fig. 1-2, and read paragraph [0076-0079])
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of HAO in view of BEIJBOM of having a method for configuring a classifier-localizer module, with the teachings of CHOI of having an aircraft comprising a system.
Wherein HAO’s method having an aircraft comprising a system according to claim 15.
The motivation behind the modification would have been to obtain a method that improves object detection and recognition, since both HAO and CHOI concern image analysis and object detection/recognition. Wherein HAO’s provides improves object recognition and detection accuracy, while CHOI’s systems and improves the accuracy of machine learning model. Please see HAO et al. (US 20210209368 A1), Paragraph [0020-0023] and CHOI et al. (US 20220051017 A1), Abstract and Paragraph [0017 and 0032].
Allowable Subject Matter
Claims 8-9 are therefrom objected to as being dependent upon rejected base claims 6-7, respectively but would be allowable if rewritten in independent form including all of the limitations of the base claims and any intervening claims once the claim objections are overcome.
The following is a statement of reasons for the indication of allowable subject matter:
PNG
media_image1.png
131
662
media_image1.png
Greyscale
Regarding claim 8, the prior arts fail to explicitly teach wherein said classification loss function is evaluated using the expression:
, where LEC is the classification loss function, M is the number of assignable classes of said list of classes, δ(C.sub.i, C.sub.r) is a binary indicator equal to 0 if for a said object the class C.sub.i of said list of classes is different from the known class C.sub.r of said object and otherwise equal to 1, and p.sub.o=(p.sub.o,i).sub.1≤i≤M with p.sub.o,i a probability determined by the classifier-localizer module that said object belongs to the class C.sub.i, as claimed in claim 8.
Regarding claim 9, the prior arts fail to explicitly teach, wherein said localization loss function is evaluated using the expression:
PNG
media_image2.png
137
709
media_image2.png
Greyscale
, where L.sub.IMSE is the localization loss function, N is the number of images of said at least one reference sequence, I=(I.sub.i).sub.1≤i≤N with ∥I∥.sub.1=Σ.sub.i|I.sub.i| and I.sub.i a binary indicator equal to 1 if a said object is present in an image of index i of said at least one reference sequence and otherwise equal to 0, P.sub.o=(P.sub.o,i).sub.1≤i≤N with P.sub.o,i the estimated position of said object determined by the classifier-localizer module for the image of index i, and P.sub.r=(P.sub.r,i).sub.1≤i≤N with P.sub.r,i the known position of said object for the image of index i, as claimed in claim 20.
Conclusion
Listed below are the prior arts made of record and not relied upon but are considered pertinent to applicant`s disclosure.
HSAIO et al. (US 20230147286 A1)- A neural network architecture and method for analysis of time series images from an image source employs a 3D-UNet convolutional neural network (CNN) configured to receive the time series images and generate spatiotemporal feature maps therefrom. Multiple sub-convolutional neural network output prongs based on an SRNet architecture receive the feature maps and simultaneously generate inferences for image segmentation, regression of values, and multi-landmark localization....…....................... Please see para. [0075-0080]. Abstract.
FAN et al. (US 20180300880 A1)- Embodiments include apparatus and methods for training and/or using a convolutional neural network. An image pyramid is calculated from an original image including at least one object of interest. The image pyramid includes a first image under analysis and a second image under analysis. Image patches are calculated at a first predetermined size relative to the first image under analysis and a second plurality of image patches having a second predetermined size relative to the second image under analysis. The convolutional neural network is trained using the image patches and subsequent images are analyzed by the convolutional neural network using similar image patches...…....................... Please see para. [0057-0061]. Abstract.
WU et al. (US 20220358328 A1)- Provided are methods for end-to-end perception system training using fused images, which can include fusing different types of images to form a fused image, extracting features from the fused image, calculating a loss, and modifying at least one network parameter of an image semantic network based on the loss. Systems and computer program products are also provided..…....................... Please see para. [0119]. Abstract.
PHAM et al. (US 20220383037 A1)- This disclosure describes one or more implementations of systems, non-transitory computer-readable media, and methods that extract multiple attributes from an object portrayed in a digital image utilizing a multi-attribute contrastive classification neural network. For example, the disclosed systems utilize a multi-attribute contrastive classification neural network that includes an embedding neural network, a localizer neural network, a multi-attention neural network, and a classifier neural network. In some cases, the disclosed systems train the multi-attribute contrastive classification neural network utilizing a multi-attribute, supervised-contrastive loss. In some embodiments, the disclosed systems generate negative attribute training labels for labeled digital images utilizing positive attribute labels that correspond to the labeled digital images........................ Please see Fig. 3-7. Abstract.
JAIN et al. (US 20220256175 A1)- A computer-implemented method for generating video representations utilizing a hierarchical video encoder includes obtaining a video, wherein the video includes a plurality of frames, processing each of the plurality of frames with a machine-learned frame-level encoder model to respectively generate a plurality of frame representations for the plurality of frames, the plurality of frame representations respective to the plurality of frames determining a plurality of segment representations representative of a plurality of video segments including one or more of the plurality of frames, the plurality of segment representations based at least in part on the plurality of frame representations, processing the plurality of segment representations with a machine-learned segment-level encoder model to generate a plurality of contextualized segment representations, determining a video representation based at least in part on the plurality of contextualized segment representations, and providing the video representation as an output........................... Please see Fig. 5 and Para. [0035-0043, 0060 and 0100-0101]. Abstract.
BHANU et al. (US 20200394413 A1)- A system and method leverages understanding of complex dribbling video clips by representing a video sequence with a single Dribble Energy Image (DEI) that is informative for dribbling styles recognition. To overcome the shortage of labelled data, a dataset of soccer video clips employs Mask-RCNN to segment out dribbling players and OpenPose to obtain joints information of dribbling players. To solve issues caused by camera motions in highlight soccer videos, the system registers a video sequence to generate a single image representation DEI and dribbling styles classification......................... Please see para. [0097-0099, 0119, 0124]. Abstract.
HAN et al. (US 20200065976 A1)- The invention disclosed here relates to a method and system for real-time target tracking based on deep learning. The method for real-time target tracking according to an embodiment is performed by a computing device including a processor, and includes pre-training a target tracking model for detecting a tracking target from an image using pre-inputted training data, receiving an image with a plurality of frames, and detecting the tracking target for each of the plurality of frames by applying the target tracking model to the image. According to an embodiment, there is a remarkable reduction in the time required to detect the target from the image, thereby allowing real-time visual tracking, and improvement of the hierarchical structure and introduction of a new loss function make it possible to achieve more precise localization and distinguish different targets of similar shapes.......................... Please see Para. [0068-0077]. Abstract.
Redmon et al. (US 20190102646 A1)- Systems and methods are disclosed for image-based object detection and classification. For example, methods may include accessing an image from an image sensor; applying a convolutional neural network to the image to obtain localization data to detect an object depicted in the image and to obtain classification data to classify the object, in which the convolutional neural network has been trained in part using training images with associated localization labels and classification labels and has been trained in part using training images with associated classification labels that lack localization labels; annotating the image based on the localization data and the classification data to obtain an annotated image; and storing, displaying, or transmitting the annotated image.............................. Please see para. [0077-0089]. Abstract.
ZHAO et al. (US 20200349394 A1)- Systems and techniques for classification and localization based on annotation information are presented. In one example, a system trains a convolutional neural network based on training data and a plurality of images. The training data is associated with a plurality of patients from at least one imaging device. The plurality of images is associated with a plurality of masks from a plurality of objects. The convolutional neural network comprises a decoder consisting of at least one up-sampling layer and at least one convolutional layer. The system also generates a loss function based on the plurality of masks, where the loss function is iteratively back propagated to tune parameters of the convolutional neural network. The system also predicts a classification label for an input image based on the convolutional neural network........................... Please see Fig. 6 and para. [0041-0045]. Abstract.
Any inquiry concerning this communication or earlier communications from the examiner
should be directed to Aaron Bonansinga whose telephone number is (703) 756-5380 The examiner can normally be reached on Monday-Friday, 9:00 a.m. - 6:00 p.m. ET.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s
supervisor, Chineyere Wills-Burns can be reached by phone at (571) 272-9752. The fax phone number for the organization where this application or proceeding is assigned is (571) 273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/AARON TIMOTHY BONANSINGA/Examiner, Art Unit 2673
/CHINEYERE WILLS-BURNS/Supervisory Patent Examiner, Art Unit 2673
Prosecution Insights