DETAILED ACTION
Continued Examination Under 37 CFR 1.114
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on April 26, 2010 has been entered.
In applicant’s reply filed on May 14, 2026, claims 1, 4, 6, 8-9, 12, 18 and 20 are currently amended. Claims 1- 20 are pending in this application.
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Response to Arguments
Applicant’s arguments, see “Remarks” filed May 14, 2026, with respect to the following have been fully considered and are persuasive.
Applicant' s amendments overcome the rejections of claims 1-20 under 35 U.S.C. 101 for being directed to non-statutory subject matter, and the rejection is hereby withdrawn.
Applicant' s amendments overcome the rejections of Claims 1-3, 5-14, and 17-120 under 35 U.S.C. § 103 as being unpatentable over Zatpeyakin et al. (US PGPub US2018/0182165 A1) in view of Riemenschneider et al. (US PGPub 20240242444, originally filed on January 17, 2023), hereby referred to as “Riemenschneider” , and the rejection is hereby withdrawn.
Applicant' s amendments overcome the rejections of, and the rejection is hereby withdrawn.
Applicant’s arguments have been fully considered and are moot in view of the new grounds of rejection as presented below, necessitated by applicant’s amendments.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all obviousness rejections set forth in this Office action:
(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are such that the subject matter as a whole would have been obvious at the time the invention was made to a person having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the manner in which the invention was made.
Claims 1-3, 5-14, and 17-120 are rejected under 35 U.S.C. § 103 as being unpatentable over Zatpeyakin et al. (US PGPub US2018/0182165 A1) in view of Machefar et al. (US PGPub US 2022/0391615 A1), hereby referred to as “Machefer”.
Consider Claims 1, 18 and 20.
Zatepyakin teaches:
1. A method comprising: / 18. A non-transitory computer-readable storage medium having stored thereon computer executable instructions, which when executed by a computing device, cause the computing device to be operable for: / 20. An apparatus comprising: one or more computer processors; and a computer-readable storage medium comprising instructions for controlling the one or more computer processors to be operable for: (Zatepyakin: abstract, A face tracking system generates a model for extracting a set of facial anchor points on a face within a portion of a face image based a multiple-level cascade of decision trees. The face tracking system identifies a mesh shape adjusted to an image of a face. For each decision tree, the face tracking system identifies an adjustment vector for the mesh shape relative to the image of the face. For each cascade level, the face tracking system combines the identified adjustment for each decision tree to determine a combined adjustment vector for the cascade level. The face tracking system modifies adjustment of the mesh shape to the face in the image based on the combined adjustment vector. The face tracking system reduces the model to a dictionary and atom weights using a learned dictionary. The model may be more easily transmitted to devices and stored on devices. [0018]-[0026], Figures 1A-B; [0018] FIG. 1A is a system environment 100 of a face tracking system 140 including a face alignment module 146, in accordance with an embodiment. The system environment 100 shown by FIG. 1 comprises one or more client devices 110, a network 120, one or more external sources 130, and the face tracking system 140. In alternative configurations, different and/or additional components may be included in the system environment 100.)
1. receiving an image; / 18. receiving an image; / 20. receiving an image; (Zatepyakin: [0019] The client devices 110 are one or more computing devices capable of capturing face images of a user, receiving user input as well as transmitting and/or receiving data via the network 120. In one embodiment, a client device 110 is a conventional computer system that includes an imaging device for capturing images having a user's face. Examples of an imaging device include a camera, a video camera, or other image capture device. [0020] The application 112 on the client device may perform facial alignment of a face within the captured image. To determine the facial image, the application 112 applies a trained model to analyze a face in the image to extract a set of facial anchor points on the face. The application 112 may receive the trained model from the face tracking system 140 and after applying the model, use the extracted set of facial anchor points to interpret or augment the image.)
1. analyzing the image based on a plurality of anchor shapes to generate respective outputs for anchor shapes in the plurality of anchor shapes, wherein the output rates a cropping of the image using a respective anchor shape; / 18. analyzing the image based on a plurality of anchor shapes to generate respective outputs for anchor shapes in the plurality of anchor shapes, wherein the output rates a cropping of the image using a respective anchor shape; / 20. analyzing the image based on a plurality of anchor shapes to generate respective outputs for anchor shapes in the plurality of anchor shapes, wherein the output rates a cropping of the image using a respective anchor shape; (Zatepyakin: [0020] The application 112 on the client device may perform facial alignment of a face within the captured image. To determine the facial image, the application 112 applies a trained model to analyze a face in the image to extract a set of facial anchor points on the face. The application 112 may receive the trained model from the face tracking system 140 and after applying the model, use the extracted set of facial anchor points to interpret or augment the image. The application 112 may determine facial anchor points as described below with respect to modules of the face tracking system 140. After identifying the facial anchor points, the application 112 may use the anchor points to track and characterize the face, for example to look for further features of the face between anchor points, or to display an overlay or mask over the user's face. The anchor points may also be captured over time to identify how a user's face moves during a video capture, which may for example be used to populate animated expressions using the anchor points, among other uses. The application 112 may also send the set of facial anchor points to another client device or the face tracking system 140 for similar uses. An a further example, the application 112 may provide video chat services for users of the client device, permitting users to capture and send video to another user. By capturing the anchor points of a face during the video, the video can be augmented using the anchor points, e.g., to add a mask to a user's face, or the by sending the anchor points for each frame of the video to another client device. In some embodiments, the anchor points may be determined for an initial frame of the video, and subsequent frames may use alternate face tracking techniques to monitor the movement of the face after the anchor points have been determined. [0021]-[0026], [0027] FIG. 1B shows examples of a captured image 160 and identification of a facial shape for the image, in accordance with an embodiment. FIG. 1B includes a bounding box 162 having an identified face 164, a cropped bounding box 166, a default shape 168 and a fitted shape 170 of the system environment illustrated in FIG. 1A. As shown in FIG. 1B, the default shape 168 has predefined facial anchor points around eyes, noses, mouth, and jaw lines. The default shape 168 is centered and scaled according to the cropped bounding box 166. The default shape does not account for the actual position and alignment of the face in the image. By applying the prediction model as described below, the fitted shape 170 is identified that has better positions of the facial anchor points aligned with the identified face in the cropped bounding box 166 than the adjusted default shape 172.)
1. analyzing respective outputs for the anchor shapes in the plurality of anchor shapes to select an anchor shape; / 18. analyzing respective outputs for the anchor shapes in the plurality of anchor shapes to select an anchor shape; / 20. analyzing respective outputs for the anchor shapes in the plurality of anchor shapes to select an anchor shape; (Zatepyakin:[0028]- n one embodiment, the face alignment module 146 uses a barycentric mesh-based shape for prediction. The barycentric mesh-based shape uses a barycentric coordinates system. The barycentric coordinate system is a coordinate system in which a position of a point within an element (e.g., a triangle, or tetrahedron) is represented by a linear combination of its vertices. For example, when the element is a triangle, points inside the triangle can be represented by a linear combination of three vertices of the triangle. The mesh-based shape may consist of multiple triangles covering all the predefined facial anchor points. Each facial anchor point can be represented by a linear combination of vertices in an associated triangle. [0029] FIG. 2 shows an example of barycentric mesh-based shapes, in accordance with an embodiment. As shown in FIG. 2, a barycentric mesh-based default shape 210 has multiple triangles. The triangles cover all the predefined facial anchor points as shown in dash lines. The barycentric mesh-based default shape 210 may be adjusted according to the cropped bounding box 166. The adjusted barycentric mesh-based default shape 220 may determine updated positions of predefined facial anchor points 230 using vertices of the associated triangles to correspond the predefined facial anchor points to the default shape applied to the cropped bounding box 166. When applying the prediction model, a barycentric mesh-based fitted shape 240 is generated to adjust the mesh to the face within the image and include updated triangles. Then, the barycentric mesh-based fitted shape 240 may determine updated positions of predefined facial anchor points 250 using vertices of associated update triangles. )
1. and cropping the image using the anchor shape. / 18. and cropping the image using the anchor shape. / 20. and cropping the image using the anchor shape. (Zatepyakin: [0031] FIG. 3 shows an example of a regression tree 300 for generating an adjustment vector, in accordance with an embodiment. In the example of FIG. 3, the regression tree 300 includes two depths and 4 leafs (N3-N6). An input for the regression tree 300 includes a cropped bounding box 168 having an identified face and a barycentric mesh-based default shape 210. In other examples, the mesh shape input to the tree may include already-applied adjustments to the default mesh, for example from a prior adjustment of the shape to match the face. For node N0, two positions A and B close to predefined facial anchor points are specified in the default shape 210. The default shape 210 is adjusted according to the cropped bounding box 168. After adjusting the default shape to the cropped bounding box 168, the adjusted default shape 220 may have the same size as the cropped bounding box 168.)
Even if Zatepyakin does not teach:
1/18/20. forming different portions of the image using respective shapes of a plurality of anchor shapes, wherein the shape of the anchor shapes form the different portions;
1/18/20. analyzing respective content within the different portions of the image and not respective content outside of the different portions of the image to generate respective outputs for anchor shapes in the plurality of anchor shapes, wherein the output rates a cropping of the image using a respective anchor shape;
1/18/20. and cropping the image using the anchor shape that is selected, wherein the cropping crops out content outside of the anchor shape from the image.
Machefar teaches:
1. A method comprising: / 18. A non-transitory computer-readable storage medium having stored thereon computer executable instructions, which when executed by a computing device, cause the computing device to be operable for: / 20. An apparatus comprising: one or more computer processors; and a computer-readable storage medium comprising instructions for controlling the one or more computer processors to be operable for: (Machefer: abstract, Aspects include methods and apparatuses generally relating to agricultural technology and artificial intelligence and, more particularly, to counting and sizing plants in a field. One aspect relates to a plants analysis apparatus for computer analysis of plants in an area of interest that generally includes an input device for receiving at least one aerial image of the area of interest; and an object-mask-predicting region-based convolutional neural network, Mask R-CNN, for performing object detection, wherein the Mask R-CNN is trained to detect a selected vegetable and to determine numbers and sizes of objects detected. [0038]-[0049], Figures 1-2, [0038] Referring to FIG. 1 , a computer implemented program 100 for analysing plants is illustrated. UAV raw images 101 may be defined as at least one unprocessed aerial image of one or more fields obtained by an unmanned aerial vehicle (UAV) during a flight. UAV flight metadata 102 is geolocation and image quality metadata related to the flight of the UAV. Field boundaries 103 specify the geographic extent of each field imaged during the flight. Field boundaries are defined and stored offline before the first UAV flight of the one or more fields. One or more fields may be imaged during a single flight. [0047] In FIG. 1 , each box is a module of computer code performed by a processor, having memory that stores parameters and training data.)
1. receiving an image; / 18. receiving an image; / 20. receiving an image; (Machefar: [0038], [0050] Referring to FIG. 2 , the images 201 are images to which the Mask R-CNN algorithm is applied. The images 201 may be RGB images and may be the non-overlapping images into which the orthomosaic is split into in block 105 of FIG. 1 .)
1/18/20. forming different portions of the image using respective shapes of a plurality of anchor shapes, wherein the shape of the anchor shapes form the different portions; (Machefar: [0053] The region proposal network (RPN) 204 is a fully convolutional network that simultaneously predicts object bounds and objectness scores at each position. In this way, the RPN 204 may propose a region in which a particular object lies. The RPN 204 may learn this from the feature maps 203 obtained from the backbone 202. [0054] Anchor boxes are a set of predefined bounding boxes of a certain height and width which are defined to capture the scale and aspect ratio of specific object classes that a user may want to detect. These predetermined boxes are designed for each feature map with specific base scales linked to the feature map shapes, and a specific ratio is applied to them. Each anchor box may be pre-configured according to a corresponding feature map and each anchor box has a base scale (width in pixels) linked to the shape of its associated feature map. Anchor boxes may be generated at each pixel of each feature map and may be separated by a specific stride, the stride being a number of pixels that equates to a downscaling factor for the original images. Anchor boxes may be obtained by a sliding window method and may be generated at each pixel of each feature map with a specific stride. RPN targets 205 are extra inputs for the RPN 204 and may be generated from a collection of anchor boxes. The RPN targets 205 can therefore be used to help the RPN 204 propose regions of interest (ROIs).)
1/18/20. analyzing respective content within the different portions of the image and not respective content outside of the different portions of the image to generate respective outputs for anchor shapes in the plurality of anchor shapes, wherein the output rates a cropping of the image using a respective anchor shape; (Machefar: [0055] An anchor box may be compared with a ground truth box to determine its accuracy. A ground truth box is provided based on empirical input. More specifically, a ground truth box is a rectangular bounding box from a testing set, which specifies where an object is in an image. Ground truth boxes are typically hand-drawn and are used for training the Mask R-CNN model. [0056] The proposal layer 206 comprises a filtering block which only keeps relevant suggestions from the RPN 204. The RPN 204 feeds into the proposal layer 206. The proposal layer 206 outputs Regions of Interest (ROIs). [0057] The training path 207 is a path that the Mask R-CNN may follow in order to train itself. The training path 207 comprises a detection target layer 208, an FPN classifier 209 and an FPN mask 210.)
1. analyzing respective outputs for the anchor shapes in the plurality of anchor shapes to select an anchor shape; / 18. analyzing respective outputs for the anchor shapes in the plurality of anchor shapes to select an anchor shape; / 20. analyzing respective outputs for the anchor shapes in the plurality of anchor shapes to select an anchor shape;(Machefar: [0058]-[0059] The FPN classifier 209 and the FPN mask graph 210 are both parts of a feature production network (FPN). An FPN is a feature extractor that takes a single-scale image of an arbitrary size as input, and outputs proportionally sized feature maps at multiple levels, in a fully convolutional fashion. The FPN classifier 209 classifies objects in the ROIs. Specifically, the FPN classifier 209 outputs a classifier head with logits and probabilities for each item of the collection to be an object and belong to a certain class as well as refined box coordinates.)
1/18/20. and cropping the image using the anchor shape that is selected, wherein the cropping crops out content outside of the anchor shape from the image. (Machefar: [0059] The FPN classifier 209 and the FPN mask graph 210 are both parts of a feature production network (FPN). An FPN is a feature extractor that takes a single-scale image of an arbitrary size as input, and outputs proportionally sized feature maps at multiple levels, in a fully convolutional fashion. The FPN classifier 209 classifies objects in the ROIs. Specifically, the FPN classifier 209 outputs a classifier head with logits and probabilities for each item of the collection to be an object and belong to a certain class as well as refined box coordinates.
[0060] The output of the FPN mask graph step 210 is a collection of masks of fixed squared size which may be re-sized to the shape in pixels of the corresponding bounding box extracted in the FPN classifier 209. [0061] The inference path 211 is a path that the Mask R-CNN algorithm, once trained, may follow in order to make predictions. The inference path 211 comprises an FPN classifier 212, a detection layer 213, and an FPN mask 214. The FPN classifier 212 functions in the same way as the FPN classifier 209, but is directly applied to ROIs extracted from the proposal layer 206.)
It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to modify Zetpayakin’s method and system for a learning algorithm for feature extraction using atom weights in a learned dictionary to leverage the image analysis algorithm of Machefar for identifying objects of interest using variable-sized anchor boxes corresponding to different shapes. The determination of obviousness is predicated upon the following findings: Both references are directed towards the field of 3D image analysis, and one skilled in the art would have been motivated to modify the feature extraction process of Zetpayakin in order to incorporate in the size-varying anchor boxes for identifying features of interest. Furthermore, the prior art collectively includes each element claimed (though not all in the same reference), and one of ordinary skill in the art could have combined the elements in the manner explained above using known engineering design, interface and programming techniques, without changing a “fundamental” operating principle of Zetpayakin, while the teaching of continues to perform the same function as originally taught prior to being combined, in order to produce the repeatable and predictable result of leveraging anchor content with the physical layout of spaces for improvement in the overall feature detection and analysis process. It is for at least the aforementioned reasons that the examiner has reached a conclusion of obviousness with respect to the claim in question.
Consider Claims 2 and 19.
The combination of Zatepyakin and Machefar teaches:
2. The method of claim 1, wherein anchor shapes in the plurality of anchor shapes crop different portions of the image. / 19. The non-transitory computer-readable storage medium of claim 18, wherein anchor shapes in the plurality of anchor shapes crop different portions of the image. (Zatepyakin: [0035]-[0044], [0037] Though the cascading trees shown in FIG. 4 include two levels for illustration, the changes to the shape from one level of the cascade to another may be applied to a large number of cascade levels and across many trees for each level with greater depth. [0038] FIG. 5 is a flowchart illustrating a process for predicting a shape based on a cascade of regression trees, in accordance with an embodiment. The process 500 may include different or additional steps than those described in conjunction with FIG. 5 in some embodiments or perform steps in different orders than the order described in conjunction with FIG. 5. [0039] The face tracking system 140 identifies 530 a mesh shape adjusted to an image of a face, the mesh shape having a set of elements, each element having a set of vertices. For example, the mesh shape is in a barycentric coordinate system. At a first cascade level, the mesh shape is a default shape or the default shape adjusted to the cropped image. An example is described in FIG. 4. In another example, at other cascade levels, the current shape is a fitted shape from the prior level. The fitted shape may have the same size as an image to be aligned as discussed in FIG. 4.)
Consider Claim 3.
The combination of Zatepyakin and Machefar teaches:
3. The method of claim 1, wherein anchor shapes in the plurality of anchor shapes are predefined shapes. (Zatepyakin: [0035]-[0044], [0037] Though the cascading trees shown in FIG. 4 include two levels for illustration, the changes to the shape from one level of the cascade to another may be applied to a large number of cascade levels and across many trees for each level with greater depth. [0038] FIG. 5 is a flowchart illustrating a process for predicting a shape based on a cascade of regression trees, in accordance with an embodiment. The process 500 may include different or additional steps than those described in conjunction with FIG. 5 in some embodiments or perform steps in different orders than the order described in conjunction with FIG. 5. [0039] The face tracking system 140 identifies 530 a mesh shape adjusted to an image of a face, the mesh shape having a set of elements, each element having a set of vertices. For example, the mesh shape is in a barycentric coordinate system. At a first cascade level, the mesh shape is a default shape or the default shape adjusted to the cropped image. An example is described in FIG. 4. In another example, at other cascade levels, the current shape is a fitted shape from the prior level. The fitted shape may have the same size as an image to be aligned as discussed in FIG. 4.)
Consider Claim 4.
The combination of Zatepyakin and Machefar teaches:
4. The method of claim 1, wherein analyzing respective content with the different portions of the image comprises: generating a feature map from the image, wherein the feature map represents one or more characteristics of the image; and analyzing the feature map to generate respective outputs for the anchor shapes. (Zatepyakin: [0032] After comparing the normalized difference of pixels C′ and D′, node N1 proceeds to node N3 or N4 based on the threshold. If the normalized difference is smaller than the first learned threshold, at a leaf N4, an adjustment vector is generated. The adjustment vector is applied to the adjusted default shape 220 to generate a fitted shape 320. The fitted shape 320 has the same size as the cropped bounding box 168. [0033] Since the positions of a node are defined with respect to the elements of the adjusted default shape, the pixel coordinates are quickly identified using the vertices of the specified element in the adjusted default shape. The barycentrically-defined position can then be applied to the vertices to determine the pixel location within the image. This permits rapid traversal of the tree, as identification of desired pixels for the threshold comparison simply looks up the location of the desired pixel by the coordinates of the adjusted default shape, which is ‘overlaid’ on the image and mapped to the image coordinates. As such, this technique does not require a transformation matrix (e.g., describing scale and rotation modifications) or other complex formula to map pixel comparison locations for a node to the image. This reduces errors and computational cost caused by calculations of transformation matrix. A Prediction Model Based on a Cascade of Regression Trees [0034] FIG. 4 shows an example of a prediction model based on a cascade 400 of regression trees, in accordance with an embodiment. In some embodiments, a prediction model may be generated by a cascade of regression trees. A cascade of regression trees may have multiple levels and multiple regression trees for each level. Machefar: [0074] Eventually, the RPN targets 205 have two components for each image: a vector which states if each of the nba anchors is positive, neutral or negative, and the second component which is represented by delta coordinates between ground truth boxes and positive anchors among the RPNtapi selected anchors to train the RPN 204. Only mGTi ground truth instances are kept per image to avoid training on images with too many objects to detect. This parameter is important for training on natural scene images composing the COCO dataset as they might contain an overwhelming number of overlapping objects. [0075] Dimensions of the targets for one image are [nba] and [RPNtapi, (dy, dx, log log (dh), log log (dw))], where dy and dx are the normalised distance of the coordinates centers between ground truth and anchor boxes, whereas log log (dh) and log log (dw) respectively deal with the logarithm delta between height and width. Finally, the RPN 204 is a FCN aiming at predicting these targets. [0076] The RPN 204 feeds into the Proposal Layer 206. The Proposal Layer 206 does not consist of a network, but a filtering block which only keeps relevant suggestions from the RPN 204. As already stated, the RPN 204 produces scores for each of the nbaanchors with the probability to be characterised as positive, neutral or negative and the Proposal Layer 206 begins by keeping the highest scores to select the best pNMSl anchors. Machefar: [0074] Eventually, the RPN targets 205 have two components for each image: a vector which states if each of the nba anchors is positive, neutral or negative, and the second component which is represented by delta coordinates between ground truth boxes and positive anchors among the RPNtapi selected anchors to train the RPN 204. Only mGTi ground truth instances are kept per image to avoid training on images with too many objects to detect. This parameter is important for training on natural scene images composing the COCO dataset as they might contain an overwhelming number of overlapping objects. [0075] Dimensions of the targets for one image are [nba] and [RPNtapi, (dy, dx, log log (dh), log log (dw))], where dy and dx are the normalised distance of the coordinates centers between ground truth and anchor boxes, whereas log log (dh) and log log (dw) respectively deal with the logarithm delta between height and width. Finally, the RPN 204 is a FCN aiming at predicting these targets. [0076] The RPN 204 feeds into the Proposal Layer 206. The Proposal Layer 206 does not consist of a network, but a filtering block which only keeps relevant suggestions from the RPN 204. As already stated, the RPN 204 produces scores for each of the nbaanchors with the probability to be characterised as positive, neutral or negative and the Proposal Layer 206 begins by keeping the highest scores to select the best pNMSl anchors.)
Consider Claim 6.
The combination of Zatepyakin and Machefar teaches:
6. The method of claim 1, wherein analyzing respective content within the different portions of the image comprises: analyzing the respective content within the different portions of the image using a plurality of prediction networks, wherein prediction networks in the plurality of prediction networks are associated with respective anchor shapes in the plurality of anchor shapes. (Zatepyakin: [0024] The interface module 142 facilitates the communication among the client device 110, the face tracking system 140, and the external source 130. In one embodiment, the interface module 142 interacts with the client devices 110 and may provide a prediction model for extracting anchor points to the client device 110, and may also receive the captured face image and provide extracted facial anchor points to the client device 110. The interface module 142 may receive one or more face databases from the external source 130. In another embodiment, the interface module 142 may provide the prediction model to the client device 110 for further processing. [0025]-[0026] The face alignment module 146 localizes facial anchor points with the captured face image using a prediction model. Examples of facial anchor points may include contour points around facial features such as eyes, noses, mouth, and jaw lines. The prediction model predicts a fitted shape of the face based on a default shape and the captured face image. A default shape provides a set of predefined facial anchor points corresponding to a generic face. In some embodiments, a default shape may be a mean shape obtained from training data as further described below. The default shape may be centered and scaled according to a bounding box including an identified face. The bounding box may be cropped for further processing to reduce computational cost. Machefar: [0074] Eventually, the RPN targets 205 have two components for each image: a vector which states if each of the nba anchors is positive, neutral or negative, and the second component which is represented by delta coordinates between ground truth boxes and positive anchors among the RPNtapi selected anchors to train the RPN 204. Only mGTi ground truth instances are kept per image to avoid training on images with too many objects to detect. This parameter is important for training on natural scene images composing the COCO dataset as they might contain an overwhelming number of overlapping objects. [0075] Dimensions of the targets for one image are [nba] and [RPNtapi, (dy, dx, log log (dh), log log (dw))], where dy and dx are the normalised distance of the coordinates centers between ground truth and anchor boxes, whereas log log (dh) and log log (dw) respectively deal with the logarithm delta between height and width. Finally, the RPN 204 is a FCN aiming at predicting these targets. [0076] The RPN 204 feeds into the Proposal Layer 206. The Proposal Layer 206 does not consist of a network, but a filtering block which only keeps relevant suggestions from the RPN 204. As already stated, the RPN 204 produces scores for each of the nbaanchors with the probability to be characterised as positive, neutral or negative and the Proposal Layer 206 begins by keeping the highest scores to select the best pNMSl anchors. Machefar: [0074] Eventually, the RPN targets 205 have two components for each image: a vector which states if each of the nba anchors is positive, neutral or negative, and the second component which is represented by delta coordinates between ground truth boxes and positive anchors among the RPNtapi selected anchors to train the RPN 204. Only mGTi ground truth instances are kept per image to avoid training on images with too many objects to detect. This parameter is important for training on natural scene images composing the COCO dataset as they might contain an overwhelming number of overlapping objects. [0075] Dimensions of the targets for one image are [nba] and [RPNtapi, (dy, dx, log log (dh), log log (dw))], where dy and dx are the normalised distance of the coordinates centers between ground truth and anchor boxes, whereas log log (dh) and log log (dw) respectively deal with the logarithm delta between height and width. Finally, the RPN 204 is a FCN aiming at predicting these targets. [0076] The RPN 204 feeds into the Proposal Layer 206. The Proposal Layer 206 does not consist of a network, but a filtering block which only keeps relevant suggestions from the RPN 204. As already stated, the RPN 204 produces scores for each of the nbaanchors with the probability to be characterised as positive, neutral or negative and the Proposal Layer 206 begins by keeping the highest scores to select the best pNMSl anchors.)
Consider Claim 7.
The combination of Zatepyakin and Machefar teaches:
7. The method of claim 6, wherein each prediction network is associated with an anchor shape in the plurality of anchor shapes and generates an output based on the respective anchor shape. (Zatepyakin: [0026] The face alignment module 146 localizes facial anchor points with the captured face image using a prediction model. Examples of facial anchor points may include contour points around facial features such as eyes, noses, mouth, and jaw lines. The prediction model predicts a fitted shape of the face based on a default shape and the captured face image. [0027] FIG. 1B shows examples of a captured image 160 and identification of a facial shape for the image, in accordance with an embodiment. FIG. 1B includes a bounding box 162 having an identified face 164, a cropped bounding box 166, a default shape 168 and a fitted shape 170 of the system environment illustrated in FIG. 1A. As shown in FIG. 1B, the default shape 168 has predefined facial anchor points around eyes, noses, mouth, and jaw lines. The default shape 168 is centered and scaled according to the cropped bounding box 166. The default shape does not account for the actual position and alignment of the face in the image. By applying the prediction model as described below, the fitted shape 170 is identified that has better positions of the facial anchor points aligned with the identified face in the cropped bounding box 166 than the adjusted default shape 172. Barycentric Mesh-Based Shape [0028] In one embodiment, the face alignment module 146 uses a barycentric mesh-based shape for prediction. The barycentric mesh-based shape uses a barycentric coordinates system. The barycentric coordinate system is a coordinate system in which a position of a point within an element (e.g., a triangle, or tetrahedron) is represented by a linear combination of its vertices. For example, when the element is a triangle, points inside the triangle can be represented by a linear combination of three vertices of the triangle. The mesh-based shape may consist of multiple triangles covering all the predefined facial anchor points. Each facial anchor point can be represented by a linear combination of vertices in an associated triangle. [0029] FIG. 2 shows an example of barycentric mesh-based shapes, in accordance with an embodiment. As shown in FIG. 2, a barycentric mesh-based default shape 210 has multiple triangles. Machefar: [0074] Eventually, the RPN targets 205 have two components for each image: a vector which states if each of the nba anchors is positive, neutral or negative, and the second component which is represented by delta coordinates between ground truth boxes and positive anchors among the RPNtapi selected anchors to train the RPN 204. Only mGTi ground truth instances are kept per image to avoid training on images with too many objects to detect. This parameter is important for training on natural scene images composing the COCO dataset as they might contain an overwhelming number of overlapping objects. [0075] Dimensions of the targets for one image are [nba] and [RPNtapi, (dy, dx, log log (dh), log log (dw))], where dy and dx are the normalised distance of the coordinates centers between ground truth and anchor boxes, whereas log log (dh) and log log (dw) respectively deal with the logarithm delta between height and width. Finally, the RPN 204 is a FCN aiming at predicting these targets. [0076] The RPN 204 feeds into the Proposal Layer 206. The Proposal Layer 206 does not consist of a network, but a filtering block which only keeps relevant suggestions from the RPN 204. As already stated, the RPN 204 produces scores for each of the nbaanchors with the probability to be characterised as positive, neutral or negative and the Proposal Layer 206 begins by keeping the highest scores to select the best pNMSl anchors.)
Consider Claim 8.
The combination of Zatepyakin and Machefar teaches:
8. The method of claim 6, wherein each prediction network analyzes information of respective content with the different portions of the image based on the respective anchor shape to generate the output. (Zatepyakin: [0027] FIG. 1B shows examples of a captured image 160 and identification of a facial shape for the image, in accordance with an embodiment. FIG. 1B includes a bounding box 162 having an identified face 164, a cropped bounding box 166, a default shape 168 and a fitted shape 170 of the system environment illustrated in FIG. 1A. As shown in FIG. 1B, the default shape 168 has predefined facial anchor points around eyes, noses, mouth, and jaw lines. The default shape 168 is centered and scaled according to the cropped bounding box 166. The default shape does not account for the actual position and alignment of the face in the image. By applying the prediction model as described below, the fitted shape 170 is identified that has better positions of the facial anchor points aligned with the identified face in the cropped bounding box 166 than the adjusted default shape 172. Barycentric Mesh-Based Shape [0028] In one embodiment, the face alignment module 146 uses a barycentric mesh-based shape for prediction. The barycentric mesh-based shape uses a barycentric coordinates system. The barycentric coordinate system is a coordinate system in which a position of a point within an element (e.g., a triangle, or tetrahedron) is represented by a linear combination of its vertices. For example, when the element is a triangle, points inside the triangle can be represented by a linear combination of three vertices of the triangle. The mesh-based shape may consist of multiple triangles covering all the predefined facial anchor points. Each facial anchor point can be represented by a linear combination of vertices in an associated triangle. [0029] FIG. 2 shows an example of barycentric mesh-based shapes, in accordance with an embodiment. As shown in FIG. 2, a barycentric mesh-based default shape 210 has multiple triangles. Machefar: [0074] Eventually, the RPN targets 205 have two components for each image: a vector which states if each of the nba anchors is positive, neutral or negative, and the second component which is represented by delta coordinates between ground truth boxes and positive anchors among the RPNtapi selected anchors to train the RPN 204. Only mGTi ground truth instances are kept per image to avoid training on images with too many objects to detect. This parameter is important for training on natural scene images composing the COCO dataset as they might contain an overwhelming number of overlapping objects. [0075] Dimensions of the targets for one image are [nba] and [RPNtapi, (dy, dx, log log (dh), log log (dw))], where dy and dx are the normalised distance of the coordinates centers between ground truth and anchor boxes, whereas log log (dh) and log log (dw) respectively deal with the logarithm delta between height and width. Finally, the RPN 204 is a FCN aiming at predicting these targets. [0076] The RPN 204 feeds into the Proposal Layer 206. The Proposal Layer 206 does not consist of a network, but a filtering block which only keeps relevant suggestions from the RPN 204. As already stated, the RPN 204 produces scores for each of the nbaanchors with the probability to be characterised as positive, neutral or negative and the Proposal Layer 206 begins by keeping the highest scores to select the best pNMSl anchors. Machefar: [0074] Eventually, the RPN targets 205 have two components for each image: a vector which states if each of the nba anchors is positive, neutral or negative, and the second component which is represented by delta coordinates between ground truth boxes and positive anchors among the RPNtapi selected anchors to train the RPN 204. Only mGTi ground truth instances are kept per image to avoid training on images with too many objects to detect. This parameter is important for training on natural scene images composing the COCO dataset as they might contain an overwhelming number of overlapping objects. [0075] Dimensions of the targets for one image are [nba] and [RPNtapi, (dy, dx, log log (dh), log log (dw))], where dy and dx are the normalised distance of the coordinates centers between ground truth and anchor boxes, whereas log log (dh) and log log (dw) respectively deal with the logarithm delta between height and width. Finally, the RPN 204 is a FCN aiming at predicting these targets. [0076] The RPN 204 feeds into the Proposal Layer 206. The Proposal Layer 206 does not consist of a network, but a filtering block which only keeps relevant suggestions from the RPN 204. As already stated, the RPN 204 produces scores for each of the nbaanchors with the probability to be characterised as positive, neutral or negative and the Proposal Layer 206 begins by keeping the highest scores to select the best pNMSl anchors.)
Consider Claim 9.
The combination of Zatepyakin and Machefar teaches:
9. The method of claim 6, wherein each prediction network analyzes information that is within the respective different portions of the image based on respective anchor shape and not outside of the respective different portions of the image to generate the output. (Zatepyakin: [0027] FIG. 1B shows examples of a captured image 160 and identification of a facial shape for the image, in accordance with an embodiment. FIG. 1B includes a bounding box 162 having an identified face 164, a cropped bounding box 166, a default shape 168 and a fitted shape 170 of the system environment illustrated in FIG. 1A. As shown in FIG. 1B, the default shape 168 has predefined facial anchor points around eyes, noses, mouth, and jaw lines. The default shape 168 is centered and scaled according to the cropped bounding box 166. The default shape does not account for the actual position and alignment of the face in the image. By applying the prediction model as described below, the fitted shape 170 is identified that has better positions of the facial anchor points aligned with the identified face in the cropped bounding box 166 than the adjusted default shape 172. Barycentric Mesh-Based Shape [0028] In one embodiment, the face alignment module 146 uses a barycentric mesh-based shape for prediction. The barycentric mesh-based shape uses a barycentric coordinates system. The barycentric coordinate system is a coordinate system in which a position of a point within an element (e.g., a triangle, or tetrahedron) is represented by a linear combination of its vertices. For example, when the element is a triangle, points inside the triangle can be represented by a linear combination of three vertices of the triangle. The mesh-based shape may consist of multiple triangles covering all the predefined facial anchor points. Each facial anchor point can be represented by a linear combination of vertices in an associated triangle. [0029] FIG. 2 shows an example of barycentric mesh-based shapes, in accordance with an embodiment. As shown in FIG. 2, a barycentric mesh-based default shape 210 has multiple triangles. Machefar: [0074] Eventually, the RPN targets 205 have two components for each image: a vector which states if each of the nba anchors is positive, neutral or negative, and the second component which is represented by delta coordinates between ground truth boxes and positive anchors among the RPNtapi selected anchors to train the RPN 204. Only mGTi ground truth instances are kept per image to avoid training on images with too many objects to detect. This parameter is important for training on natural scene images composing the COCO dataset as they might contain an overwhelming number of overlapping objects. [0075] Dimensions of the targets for one image are [nba] and [RPNtapi, (dy, dx, log log (dh), log log (dw))], where dy and dx are the normalised distance of the coordinates centers between ground truth and anchor boxes, whereas log log (dh) and log log (dw) respectively deal with the logarithm delta between height and width. Finally, the RPN 204 is a FCN aiming at predicting these targets. [0076] The RPN 204 feeds into the Proposal Layer 206. The Proposal Layer 206 does not consist of a network, but a filtering block which only keeps relevant suggestions from the RPN 204. As already stated, the RPN 204 produces scores for each of the nbaanchors with the probability to be characterised as positive, neutral or negative and the Proposal Layer 206 begins by keeping the highest scores to select the best pNMSl anchors.)
Consider Claim 10.
The combination of Zatepyakin and Machefar teaches:
10. The method of claim 9, wherein the information comprises a portion of a feature map that represents one or more characteristics of the image. (Zatepyakin: [0032] After comparing the normalized difference of pixels C′ and D′, node N1 proceeds to node N3 or N4 based on the threshold. If the normalized difference is smaller than the first learned threshold, at a leaf N4, an adjustment vector is generated. The adjustment vector is applied to the adjusted default shape 220 to generate a fitted shape 320. The fitted shape 320 has the same size as the cropped bounding box 168. [0033] Since the positions of a node are defined with respect to the elements of the adjusted default shape, the pixel coordinates are quickly identified using the vertices of the specified element in the adjusted default shape. The barycentrically-defined position can then be applied to the vertices to determine the pixel location within the image. This permits rapid traversal of the tree, as identification of desired pixels for the threshold comparison simply looks up the location of the desired pixel by the coordinates of the adjusted default shape, which is ‘overlaid’ on the image and mapped to the image coordinates. As such, this technique does not require a transformation matrix (e.g., describing scale and rotation modifications) or other complex formula to map pixel comparison locations for a node to the image. This reduces errors and computational cost caused by calculations of transformation matrix. A Prediction Model Based on a Cascade of Regression Trees [0034] FIG. 4 shows an example of a prediction model based on a cascade 400 of regression trees, in accordance with an embodiment. In some embodiments, a prediction model may be generated by a cascade of regression trees. A cascade of regression trees may have multiple levels and multiple regression trees for each level.)
Consider Claim 11.
The combination of Zatepyakin and Machefar teaches:
11. The method of claim 1, wherein the output represents a score for an overlap of a respective anchor shape and a preferred cropped image. (Zatepayakin: [0053] The compression may be performed by the face tracking system 140 by transforming the adjustment vectors of each leaf to correspond to a dictionary of “atoms.” Each atom in the dictionary describes a function to adjust the values of one or more adjustment values. Thus, rather than defining the adjustment vector of a leaf node by the complete set of vector adjustment values, the leaf node may specify a set of atoms in the dictionary and a weight for each atom in dictionary. [0054] The face tracking system identifies a dictionary of atoms for which to determine the atoms for a leaf. The dictionary of atoms may be defined by a matrix specifying functions and an adjustment value that is the primary adjustment value that the function is applied on. For example, a function may specify modifying the primary adjustment value and a set of nearby adjustment values according to a decaying function. By specifying a variety of functions and that can each apply different changes to the adjustment values and variously adjust other adjustment values, each atom may represent a significant amount of information about the adjustment values, and a small number of atoms together can represent significant change in the adjustment vector. Thus, in one embodiment, the dictionary defines a matrix in which one side of the matrix represents a set of functions and another side of the matrix represents the set of adjustment values. The intersection of a given adjustment value and a given function in the matrix represents an atom for applying the given function to the given adjustment value as the primary adjustment value. In one embodiment, there are 136 adjustment values in the matrix and 1024 functions. [0055] Machefar: [0074] Eventually, the RPN targets 205 have two components for each image: a vector which states if each of the nba anchors is positive, neutral or negative, and the second component which is represented by delta coordinates between ground truth boxes and positive anchors among the RPNtapi selected anchors to train the RPN 204. Only mGTi ground truth instances are kept per image to avoid training on images with too many objects to detect. This parameter is important for training on natural scene images composing the COCO dataset as they might contain an overwhelming number of overlapping objects. [0075] Dimensions of the targets for one image are [nba] and [RPNtapi, (dy, dx, log log (dh), log log (dw))], where dy and dx are the normalised distance of the coordinates centers between ground truth and anchor boxes, whereas log log (dh) and log log (dw) respectively deal with the logarithm delta between height and width. Finally, the RPN 204 is a FCN aiming at predicting these targets. [0076] The RPN 204 feeds into the Proposal Layer 206. The Proposal Layer 206 does not consist of a network, but a filtering block which only keeps relevant suggestions from the RPN 204. As already stated, the RPN 204 produces scores for each of the nbaanchors with the probability to be characterised as positive, neutral or negative and the Proposal Layer 206 begins by keeping the highest scores to select the best pNMSl anchors.)
Consider Claim 12.
The combination of Zatepyakin and Machefar teaches:
12. The method of claim 1, further comprising: analyzing the respective content with the different portions of the image based on the plurality of anchor shapes to generate offset coordinates for anchor shapes in the plurality of anchor shapes, wherein the offset coordinates are used to crop the image. (Zatepyakin: Node Split Based on a Pixel Comparison in a Regression Tree [0030] To extract and generate the set of anchor points 250, the prediction model uses regression trees. A regression tree has multiple nodes. The nodes can be divided into split nodes and leafs. Each leaf (e.g., a node without children) generates an adjustment vector to adjust a current shape. A split node represents a traversal decision of the tree. At each split node in the regression tree, a traversal decision is made based on a threshold difference between intensities of two pixels in a captured image. Two pixels are defined in a coordinate system of the default shape. To compare coordinates for traversing the tree, however, the coordinate system of the two pixels is translated to the location of the shape on the image. Thus, the coordinate system of the default shape is translated through the current position of the shape to determine a coordinate on the image. For example, a captured image is represented in a Cartesian coordinate system, and a barycentric mesh-based default shape is represented in a Barycentric coordinate system. Two positions in the barycentric mesh-based default shape close to predefined facial anchor points are selected. As mentioned above, the barycentric mesh-based default shape is adjusted according to a bounding box in the captured image. The shape may be further adjusted to one or more fitted shapes, as further discussed below, that closer align the shape with the facial image. The two positions of pixels in the coordinate system of the default shape are also translated according to the adjusted shape to determine the corresponding pixels on the image. A difference between intensities of the two determined pixels on the image can be calculated. For example, assume that there are two pixels A and B. A normalized difference between two pixels is calculated based on (pixel A−pixel B)/(pixel A+pixel B). In another example, a difference may be calculated based on (pixel A−pixel B). By comparing the calculated normalized difference or difference with an associated threshold, a decision is made designating a subsequent node in the tree. [0031] FIG. 3 shows an example of a regression tree 300 for generating an adjustment vector, in accordance with an embodiment. In the example of FIG. 3, the regression tree 300 includes two depths and 4 leafs (N3-N6). An input for the regression tree 300 includes a cropped bounding box 168 having an identified face and a barycentric mesh-based default shape 210. In other examples, the mesh shape input to the tree may include already-applied adjustments to the default mesh, for example from a prior adjustment of the shape to match the face. For node N0, two positions A and B close to predefined facial anchor points are specified in the default shape 210. The default shape 210 is adjusted according to the cropped bounding box 168. After adjusting the default shape to the cropped bounding box 168, the adjusted default shape 220 may have the same size as the cropped bounding box 168. Accordingly, the two positions A and B are adjusted to determine two pixels A′ and B′ in the adjusted default shape 220. Since the positions A, B may be defined with respect to a specific triangle or element in the default shape 210 and the adjusted default shape 220 is located on the image, the pixels A′, B′ in the image may be identified as the pixel location in the image corresponding to the element-defined coordinate of A′, B′ in the adjusted default shape 220. At node N0, a normalized difference between two pixels A′ and B′ in the image is calculated, and the normalized difference is compared with a first threshold associated with N0. The first threshold may be learned from training data. If the normalized difference is larger than the first learned threshold, the decision tree proceeds to node N1, and if the difference smaller than the first learned threshold, the decision tree proceeds to node N2. At a node N1, two pixels C′ and D′ close to predefined facial anchor points are similarly identified based on specified positions C and D for the node N1. That is, positions C and D may be specified by node N1 in respective barycentric coordinates with respect to an element of a mesh, and pixels C′ and D′ are determined by identifying the pixel in the image corresponding to the coordinates as applied to the location of element in the adjusted default shape 220.)
Consider Claim 13.
The combination of Zatepyakin and Machefar teaches:
13. The method of claim 12, wherein: the offset coordinates adjust coordinates of the anchor shape to generate adjusted coordinates, and the adjusted coordinates are used to crop the image. (Zatepyakin: [0032] After comparing the normalized difference of pixels C′ and D′, node N1 proceeds to node N3 or N4 based on the threshold. If the normalized difference is smaller than the first learned threshold, at a leaf N4, an adjustment vector is generated. The adjustment vector is applied to the adjusted default shape 220 to generate a fitted shape 320. The fitted shape 320 has the same size as the cropped bounding box 168. [0033] Since the positions of a node are defined with respect to the elements of the adjusted default shape, the pixel coordinates are quickly identified using the vertices of the specified element in the adjusted default shape. The barycentrically-defined position can then be applied to the vertices to determine the pixel location within the image. This permits rapid traversal of the tree, as identification of desired pixels for the threshold comparison simply looks up the location of the desired pixel by the coordinates of the adjusted default shape, which is ‘overlaid’ on the image and mapped to the image coordinates. As such, this technique does not require a transformation matrix (e.g., describing scale and rotation modifications) or other complex formula to map pixel comparison locations for a node to the image. This reduces errors and computational cost caused by calculations of transformation matrix. A Prediction Model Based on a Cascade of Regression Trees [0034] FIG. 4 shows an example of a prediction model based on a cascade 400 of regression trees, in accordance with an embodiment. In some embodiments, a prediction model may be generated by a cascade of regression trees. A cascade of regression trees may have multiple levels and multiple regression trees for each level.)
Consider Claim 14.
The combination of Zatepyakin and Machefar teaches:
14. The method of claim 1, further comprising: training a model for the plurality of anchor shapes using a training image, wherein parameters of the model are adjusted using a comparison of a first output for an anchor shape to a second output that is based on a labeled shape for the training image. (Zatepyakin: [0027] FIG. 1B shows examples of a captured image 160 and identification of a facial shape for the image, in accordance with an embodiment. FIG. 1B includes a bounding box 162 having an identified face 164, a cropped bounding box 166, a default shape 168 and a fitted shape 170 of the system environment illustrated in FIG. 1A. As shown in FIG. 1B, the default shape 168 has predefined facial anchor points around eyes, noses, mouth, and jaw lines. The default shape 168 is centered and scaled according to the cropped bounding box 166. The default shape does not account for the actual position and alignment of the face in the image. By applying the prediction model as described below, the fitted shape 170 is identified that has better positions of the facial anchor points aligned with the identified face in the cropped bounding box 166 than the adjusted default shape 172. Barycentric Mesh-Based Shape [0028] In one embodiment, the face alignment module 146 uses a barycentric mesh-based shape for prediction. The barycentric mesh-based shape uses a barycentric coordinates system. The barycentric coordinate system is a coordinate system in which a position of a point within an element (e.g., a triangle, or tetrahedron) is represented by a linear combination of its vertices. For example, when the element is a triangle, points inside the triangle can be represented by a linear combination of three vertices of the triangle. The mesh-based shape may consist of multiple triangles covering all the predefined facial anchor points. Each facial anchor point can be represented by a linear combination of vertices in an associated triangle. [0029] FIG. 2 shows an example of barycentric mesh-based shapes, in accordance with an embodiment. As shown in FIG. 2, a barycentric mesh-based default shape 210 has multiple triangles. The triangles cover all the predefined facial anchor points as shown in dash lines. The barycentric mesh-based default shape 210 may be adjusted according to the cropped bounding box 166. The adjusted barycentric mesh-based default shape 220 may determine updated positions of predefined facial anchor points 230 using vertices of the associated triangles to correspond the predefined facial anchor points to the default shape applied to the cropped bounding box 166. When applying the prediction model, a barycentric mesh-based fitted shape 240 is generated to adjust the mesh to the face within the image and include updated triangles. Then, the barycentric mesh-based fitted shape 240 may determine updated positions of predefined facial anchor points 250 using vertices of associated update triangles.)
Consider Claim 17.
The combination of Zatepyakin and Machefar teaches:
17. The method of claim 1, wherein: the anchor shape is a shape that is defined by coordinates, and the coordinates are used to crop the image. (Zatepyakin: Node Split Based on a Pixel Comparison in a Regression Tree [0030] To extract and generate the set of anchor points 250, the prediction model uses regression trees. A regression tree has multiple nodes. The nodes can be divided into split nodes and leafs. Each leaf (e.g., a node without children) generates an adjustment vector to adjust a current shape. A split node represents a traversal decision of the tree. At each split node in the regression tree, a traversal decision is made based on a threshold difference between intensities of two pixels in a captured image. Two pixels are defined in a coordinate system of the default shape. To compare coordinates for traversing the tree, however, the coordinate system of the two pixels is translated to the location of the shape on the image. Thus, the coordinate system of the default shape is translated through the current position of the shape to determine a coordinate on the image. For example, a captured image is represented in a Cartesian coordinate system, and a barycentric mesh-based default shape is represented in a Barycentric coordinate system. Two positions in the barycentric mesh-based default shape close to predefined facial anchor points are selected. As mentioned above, the barycentric mesh-based default shape is adjusted according to a bounding box in the captured image. The shape may be further adjusted to one or more fitted shapes, as further discussed below, that closer align the shape with the facial image. The two positions of pixels in the coordinate system of the default shape are also translated according to the adjusted shape to determine the corresponding pixels on the image. A difference between intensities of the two determined pixels on the image can be calculated. For example, assume that there are two pixels A and B. A normalized difference between two pixels is calculated based on (pixel A−pixel B)/(pixel A+pixel B). In another example, a difference may be calculated based on (pixel A−pixel B). By comparing the calculated normalized difference or difference with an associated threshold, a decision is made designating a subsequent node in the tree. [0031] FIG. 3 shows an example of a regression tree 300 for generating an adjustment vector, in accordance with an embodiment. In the example of FIG. 3, the regression tree 300 includes two depths and 4 leafs (N3-N6). An input for the regression tree 300 includes a cropped bounding box 168 having an identified face and a barycentric mesh-based default shape 210. In other examples, the mesh shape input to the tree may include already-applied adjustments to the default mesh, for example from a prior adjustment of the shape to match the face. For node N0, two positions A and B close to predefined facial anchor points are specified in the default shape 210. The default shape 210 is adjusted according to the cropped bounding box 168. After adjusting the default shape to the cropped bounding box 168, the adjusted default shape 220 may have the same size as the cropped bounding box 168. Accordingly, the two positions A and B are adjusted to determine two pixels A′ and B′ in the adjusted default shape 220. Since the positions A, B may be defined with respect to a specific triangle or element in the default shape 210 and the adjusted default shape 220 is located on the image, the pixels A′, B′ in the image may be identified as the pixel location in the image corresponding to the element-defined coordinate of A′, B′ in the adjusted default shape 220. At node N0, a normalized difference between two pixels A′ and B′ in the image is calculated, and the normalized difference is compared with a first threshold associated with N0. The first threshold may be learned from training data. If the normalized difference is larger than the first learned threshold, the decision tree proceeds to node N1, and if the difference smaller than the first learned threshold, the decision tree proceeds to node N2. At a node N1, two pixels C′ and D′ close to predefined facial anchor points are similarly identified based on specified positions C and D for the node N1. That is, positions C and D may be specified by node N1 in respective barycentric coordinates with respect to an element of a mesh, and pixels C′ and D′ are determined by identifying the pixel in the image corresponding to the coordinates as applied to the location of element in the adjusted default shape 220.)
Claims 4-5, and 14-16, are further rejected under 35 U.S.C. 103 as being unpatentable over Zatpeyakin et al. (US PGPub US2018/0182165 A1) in view of Machefar et al. (US PGPub US 2022/0391615 A1), hereby referred to as “Machefer”, further in view of Chen et al. (US PGPub 20230186439), hereby referred to as “Chen”.
Consider Claims 4-5.
The combination of Zatepyakin and Machefar does teach:
4. The method of claim 1, wherein analyzing respective content within the image comprises: generating a feature map from the image, wherein the feature map represents one or more characteristics of the image; and analyzing the feature map to generate respective outputs for the anchor shapes. (Zatepyakin: [0032] After comparing the normalized difference of pixels C′ and D′, node N1 proceeds to node N3 or N4 based on the threshold. If the normalized difference is smaller than the first learned threshold, at a leaf N4, an adjustment vector is generated. The adjustment vector is applied to the adjusted default shape 220 to generate a fitted shape 320. The fitted shape 320 has the same size as the cropped bounding box 168. [0033] Since the positions of a node are defined with respect to the elements of the adjusted default shape, the pixel coordinates are quickly identified using the vertices of the specified element in the adjusted default shape. The barycentrically-defined position can then be applied to the vertices to determine the pixel location within the image. This permits rapid traversal of the tree, as identification of desired pixels for the threshold comparison simply looks up the location of the desired pixel by the coordinates of the adjusted default shape, which is ‘overlaid’ on the image and mapped to the image coordinates. As such, this technique does not require a transformation matrix (e.g., describing scale and rotation modifications) or other complex formula to map pixel comparison locations for a node to the image. This reduces errors and computational cost caused by calculations of transformation matrix. A Prediction Model Based on a Cascade of Regression Trees [0034] FIG. 4 shows an example of a prediction model based on a cascade 400 of regression trees, in accordance with an embodiment. In some embodiments, a prediction model may be generated by a cascade of regression trees. A cascade of regression trees may have multiple levels and multiple regression trees for each level. Machefar: [0074] Eventually, the RPN targets 205 have two components for each image: a vector which states if each of the nba anchors is positive, neutral or negative, and the second component which is represented by delta coordinates between ground truth boxes and positive anchors among the RPNtapi selected anchors to train the RPN 204. Only mGTi ground truth instances are kept per image to avoid training on images with too many objects to detect. This parameter is important for training on natural scene images composing the COCO dataset as they might contain an overwhelming number of overlapping objects. [0075] Dimensions of the targets for one image are [nba] and [RPNtapi, (dy, dx, log log (dh), log log (dw))], where dy and dx are the normalised distance of the coordinates centers between ground truth and anchor boxes, whereas log log (dh) and log log (dw) respectively deal with the logarithm delta between height and width. Finally, the RPN 204 is a FCN aiming at predicting these targets. [0076] The RPN 204 feeds into the Proposal Layer 206. The Proposal Layer 206 does not consist of a network, but a filtering block which only keeps relevant suggestions from the RPN 204. As already stated, the RPN 204 produces scores for each of the nbaanchors with the probability to be characterised as positive, neutral or negative and the Proposal Layer 206 begins by keeping the highest scores to select the best pNMSl anchors.)
Even if the combination of Zatepyakin and Machefar does not teach: 5. The method of claim 4, wherein: the feature map comprises multiple channels, wherein channels are associated with characteristics of the image, and the channels are analyzed to generate the output.
Chen teaches:
1. A method comprising: / 18. A non-transitory computer-readable storage medium having stored thereon computer executable instructions, which when executed by a computing device, cause the computing device to be operable for: / 20. An apparatus comprising: one or more computer processors; and a computer-readable storage medium comprising instructions for controlling the one or more computer processors to be operable for: (Chen: abstract; A lane detection method integratedly using image enhancement and a deep convolutional neural network. On the assumption that lanes have similar widths in a local region of an image and a lane can be segmented into several image blocks, each of which contains lane marking in the center, a method based on a deep convolutional neural network is provided to detect lane marking blocks in the image. Input to the model includes road images captured by a camera as well as a set of enhanced images generated by the contrast limited adaptive histogram equalization (CLAHE) algorithm. The method according to the present disclosure can effectively overcome difficulties of lane detection under complex imaging conditions, such as poor image quality, and small lane marking targets, so as to achieve better robustness. [0004]-[0019])
1. receiving an image; / 18. receiving an image; / 20. receiving an image; (Chen: [0004]-[0006], Step (1), acquiring a color image I contains lanes, including three component images I(0), I(1), and I(2) corresponding to red, green, and blue color components of I, respectively; performing the CLAHE algorithm to enhance the contrast of I and generate K enhanced images, where the kth enhanced image, k = 0, 1, ..., K - 1, is formed by using the cth channel image I(c) as the input, where c is the remainder of k divided by 3.)
1. analyzing the image based on a plurality of anchor shapes to generate respective outputs for anchor shapes in the plurality of anchor shapes, wherein the output rates a cropping of the image using a respective anchor shape; / 18. analyzing the image based on a plurality of anchor shapes to generate respective outputs for anchor shapes in the plurality of anchor shapes, wherein the output rates a cropping of the image using a respective anchor shape; / 20. analyzing the image based on a plurality of anchor shapes to generate respective outputs for anchor shapes in the plurality of anchor shapes, wherein the output rates a cropping of the image using a respective anchor shape; (Chen: [0006]-[0007] Step (2), constructing the deep convolutional neural network, which consists of an input module, a spatial attention module, a feature extraction module, and a detection module, for lane detection, and stacking the three component images of the color image as well as the K enhanced images generated by the CLAHE algorithm in step (1) as a tensor including K + 3 channels to serve as the input to the deep convolutional neural network. [0027] Step (1), I is set as a to-be-processed color image, including three component images I(0), I(1), and I(2), corresponding to red, green, and blue, respectively, and the CLAHE is performed K times on I to enhance the contrast of an input image and generate K enhanced images, where the kth enhanced image, k = 0, 1, ..., K - 1, is formed by using the cth channel image I(c) as the input. In one embodiment of the present disclosure, K = 6, and c is equal to the remainder of k divided by 3. Steps of the algorithm are as follows. First, an image I(c) is processed by using a sliding window. The height and the width of the sliding window are Mb + kΔ and Nb + kΔ, respectively, where Mb, Nb, and Δ are preset constants, which may be Mb = 18, Nb = 24, and Δ = 4. Second, the histogram of a block image covered by the sliding window is calculated and denoted as H; and if any histogram bin Hi exceeds a specified limit h, it is clipped as Hi = h, and amplitude differences are accumulated according to the following formula:
PNG
media_image1.png
42
220
media_image1.png
Greyscale
; [0028]-[0031] Step (3), the deep convolutional neural network for lane detection includes an input module, a spatial attention module, a feature extraction module, and a detection module. According to the data flow of the input module during forward propagation, input data first passes through a convolutional layer with 64 7 × 7 kernels and a stride of 2, and then a batch normalization operation and a ReLU activation operation are performed. The final part of the input module is a max pooling layer with a 3 × 3 sampling kernel and with a stride of 2. [0032] Step (4), output x of the input module is an M1 × N1 × C feature map, where M1 and N1 denote the height and the width, respectively, and C denotes the number of channels of the feature map)
1. analyzing respective outputs for the anchor shapes in the plurality of anchor shapes to select an anchor shape; / 18. analyzing respective outputs for the anchor shapes in the plurality of anchor shapes to select an anchor shape; / 20. analyzing respective outputs for the anchor shapes in the plurality of anchor shapes to select an anchor shape; (Chen: [0033] Step (5), elements in the spatial attention map are taken as weights. Values of all positions of each channel of the output feature map x of the input module are multiplied by weights of corresponding positions of the spatial attention map to form a feature map, and then is fed to the feature extraction module in the embodiment of the present disclosure. [0034] Step (6), Stage 2, Stage 3, and Stage 4 convolutional layer groups of ResNet50 are taken as the feature extraction module, and the output of Stage 3 serves as the input to Stage 4 as well as the input to a convolutional layer consists of 5nB kernels of size 1 × 1 and with a stride of 1, where nB denotes a preset number of detection boxes for each anchor point, and the convolutional layer finally outputs a feature map denoted by F1. Output of Stage 4 passes through a convolutional layer consists of 5nB kernels of size 1 × 1 and with a stride of 1, and the generated feature map is up-sampled and then sums corresponding elements one by one with F1 to generate an M2 × N2 × 5nB feature map F.)
5. wherein: the feature map comprises multiple channels, wherein channels are associated with characteristics of the image, and the channels are analyzed to generate the output. (Chen: [0006]-[0007] Step (2), constructing the deep convolutional neural network, which consists of an input module, a spatial attention module, a feature extraction module, and a detection module, for lane detection, and stacking the three component images of the color image as well as the K enhanced images generated by the CLAHE algorithm in step (1) as a tensor including K + 3 channels to serve as the input to the deep convolutional neural network. [0027] Step (1), I is set as a to-be-processed color image, including three component images I(0), I(1), and I(2), corresponding to red, green, and blue, respectively, and the CLAHE is performed K times on I to enhance the contrast of an input image and generate K enhanced images, where the kth enhanced image, k = 0, 1, ..., K - 1, is formed by using the cth channel image I(c) as the input. In one embodiment of the present disclosure, K = 6, and c is equal to the remainder of k divided by 3. Steps of the algorithm are as follows. First, an image I(c) is processed by using a sliding window. The height and the width of the sliding window are Mb + kΔ and Nb + kΔ, respectively, where Mb, Nb, and Δ are preset constants, which may be Mb = 18, Nb = 24, and Δ = 4. Second, the histogram of a block image covered by the sliding window is calculated and denoted as H; and if any histogram bin Hi exceeds a specified limit h, it is clipped as Hi = h, and amplitude differences are accumulated according to the following formula:
PNG
media_image1.png
42
220
media_image1.png
Greyscale
; [0028]-[0031] Step (3), the deep convolutional neural network for lane detection includes an input module, a spatial attention module, a feature extraction module, and a detection module. According to the data flow of the input module during forward propagation, input data first passes through a convolutional layer with 64 7 × 7 kernels and a stride of 2, and then a batch normalization operation and a ReLU activation operation are performed. The final part of the input module is a max pooling layer with a 3 × 3 sampling kernel and with a stride of 2. [0032] Step (4), output x of the input module is an M1 × N1 × C feature map, where M1 and N1 denote the height and the width, respectively, and C denotes the number of channels of the feature map)
5. The method of claim 4, wherein: the feature map comprises multiple channels, wherein channels are associated with characteristics of the image, and the channels are analyzed to generate the respective output. (Chen: [0006]-[0007] Step (2), constructing the deep convolutional neural network, which consists of an input module, a spatial attention module, a feature extraction module, and a detection module, for lane detection, and stacking the three component images of the color image as well as the K enhanced images generated by the CLAHE algorithm in step (1) as a tensor including K + 3 channels to serve as the input to the deep convolutional neural network. [0027] Step (1), I is set as a to-be-processed color image, including three component images I(0), I(1), and I(2), corresponding to red, green, and blue, respectively, and the CLAHE is performed K times on I to enhance the contrast of an input image and generate K enhanced images, where the kth enhanced image, k = 0, 1, ..., K - 1, is formed by using the cth channel image I(c) as the input. In one embodiment of the present disclosure, K = 6, and c is equal to the remainder of k divided by 3. Steps of the algorithm are as follows. First, an image I(c) is processed by using a sliding window. The height and the width of the sliding window are Mb + kΔ and Nb + kΔ, respectively, where Mb, Nb, and Δ are preset constants, which may be Mb = 18, Nb = 24, and Δ = 4. Second, the histogram of a block image covered by the sliding window is calculated and denoted as H; and if any histogram bin Hi exceeds a specified limit h, it is clipped as Hi = h, and amplitude differences are accumulated according to the following formula:
PNG
media_image1.png
42
220
media_image1.png
Greyscale
; [0028]-[0031] Step (3), the deep convolutional neural network for lane detection includes an input module, a spatial attention module, a feature extraction module, and a detection module. According to the data flow of the input module during forward propagation, input data first passes through a convolutional layer with 64 7 × 7 kernels and a stride of 2, and then a batch normalization operation and a ReLU activation operation are performed. The final part of the input module is a max pooling layer with a 3 × 3 sampling kernel and with a stride of 2. [0032] Step (4), output x of the input module is an M1 × N1 × C feature map, where M1 and N1 denote the height and the width, respectively, and C denotes the number of channels of the feature map)
It would have been obvious before the effective filing date of the claimed invention was made to one of ordinary skill in the art to modify the machine learning algorithm of the combination of Zatpeyakin and Machefar’s for extraction of facial features using set of adjustable mesh shapes and anchor content with the teachings of Chen for machine learning and feature extraction of using image enhancement and contrast limited adaptive histogram equalization. The determination of obviousness is predicated upon the following findings: One skilled in the art would have been motivated to modify the combination of Zatpeyakin and Machefar in order to improve the overall feature detection and machine learning algorithms to leverage a contrast adaptive histogram equalization algorithm to ensure enhanced accuracy and robustness. Furthermore, the prior art collectively includes each element claimed (though not all in the same reference), and one of ordinary skill in the art could have combined the elements in the manner explained above using known engineering design, interface and/or programming techniques, without changing a “fundamental” operating principle of the combination of Zatpeyakin and Machefar, while the teaching of Chen continues to perform the same function as originally taught prior to being combined, in order to produce the repeatable and predictable result of enhancing overall computational efficiency and accuracy. It is for at least the aforementioned reasons that the examiner has reached a conclusion of obviousness with respect to the claim in question.
Consider Claims 14-15.
The combination of Zatpeyakin and Machefar does teach:
14. The method of claim 1, further comprising: training a model for the plurality of anchor shapes using a training image, wherein parameters of the model are adjusted using a comparison of a first output for an anchor shape to a second output that is based on a labeled shape for the training image. (Zatepyakin: [0027] FIG. 1B shows examples of a captured image 160 and identification of a facial shape for the image, in accordance with an embodiment. FIG. 1B includes a bounding box 162 having an identified face 164, a cropped bounding box 166, a default shape 168 and a fitted shape 170 of the system environment illustrated in FIG. 1A. As shown in FIG. 1B, the default shape 168 has predefined facial anchor points around eyes, noses, mouth, and jaw lines. The default shape 168 is centered and scaled according to the cropped bounding box 166. The default shape does not account for the actual position and alignment of the face in the image. By applying the prediction model as described below, the fitted shape 170 is identified that has better positions of the facial anchor points aligned with the identified face in the cropped bounding box 166 than the adjusted default shape 172. Barycentric Mesh-Based Shape [0028] In one embodiment, the face alignment module 146 uses a barycentric mesh-based shape for prediction. The barycentric mesh-based shape uses a barycentric coordinates system. The barycentric coordinate system is a coordinate system in which a position of a point within an element (e.g., a triangle, or tetrahedron) is represented by a linear combination of its vertices. For example, when the element is a triangle, points inside the triangle can be represented by a linear combination of three vertices of the triangle. The mesh-based shape may consist of multiple triangles covering all the predefined facial anchor points. Each facial anchor point can be represented by a linear combination of vertices in an associated triangle. [0029] FIG. 2 shows an example of barycentric mesh-based shapes, in accordance with an embodiment. As shown in FIG. 2, a barycentric mesh-based default shape 210 has multiple triangles. The triangles cover all the predefined facial anchor points as shown in dash lines. The barycentric mesh-based default shape 210 may be adjusted according to the cropped bounding box 166. The adjusted barycentric mesh-based default shape 220 may determine updated positions of predefined facial anchor points 230 using vertices of the associated triangles to correspond the predefined facial anchor points to the default shape applied to the cropped bounding box 166. When applying the prediction model, a barycentric mesh-based fitted shape 240 is generated to adjust the mesh to the face within the image and include updated triangles. Then, the barycentric mesh-based fitted shape 240 may determine updated positions of predefined facial anchor points 250 using vertices of associated update triangles. Machefar: [0074] Eventually, the RPN targets 205 have two components for each image: a vector which states if each of the nba anchors is positive, neutral or negative, and the second component which is represented by delta coordinates between ground truth boxes and positive anchors among the RPNtapi selected anchors to train the RPN 204. Only mGTi ground truth instances are kept per image to avoid training on images with too many objects to detect. This parameter is important for training on natural scene images composing the COCO dataset as they might contain an overwhelming number of overlapping objects. [0075] Dimensions of the targets for one image are [nba] and [RPNtapi, (dy, dx, log log (dh), log log (dw))], where dy and dx are the normalised distance of the coordinates centers between ground truth and anchor boxes, whereas log log (dh) and log log (dw) respectively deal with the logarithm delta between height and width. Finally, the RPN 204 is a FCN aiming at predicting these targets. [0076] The RPN 204 feeds into the Proposal Layer 206. The Proposal Layer 206 does not consist of a network, but a filtering block which only keeps relevant suggestions from the RPN 204. As already stated, the RPN 204 produces scores for each of the nbaanchors with the probability to be characterised as positive, neutral or negative and the Proposal Layer 206 begins by keeping the highest scores to select the best pNMSl anchors.)
15. The method of claim 14, wherein training comprises: receiving the training image and the labeled shape; (Zatepyakin: [0020] The application 112 on the client device may perform facial alignment of a face within the captured image. To determine the facial image, the application 112 applies a trained model to analyze a face in the image to extract a set of facial anchor points on the face. The application 112 may receive the trained model from the face tracking system 140 and after applying the model, use the extracted set of facial anchor points to interpret or augment the image. The application 112 may determine facial anchor points as described below with respect to modules of the face tracking system 140. After identifying the facial anchor points, the application 112 may use the anchor points to track and characterize the face, for example to look for further features of the face between anchor points, or to display an overlay or mask over the user's face. The anchor points may also be captured over time to identify how a user's face moves during a video capture, which may for example be used to populate animated expressions using the anchor points, among other uses. The application 112 may also send the set of facial anchor points to another client device or the face tracking system 140 for similar uses. An a further example, the application 112 may provide video chat services for users of the client device, permitting users to capture and send video to another user. By capturing the anchor points of a face during the video, the video can be augmented using the anchor points, e.g., to add a mask to a user's face, or the by sending the anchor points for each frame of the video to another client device. In some embodiments, the anchor points may be determined for an initial frame of the video, and subsequent frames may use alternate face tracking techniques to monitor the movement of the face after the anchor points have been determined. [0021]-[0027]; Machefar: [0074] Eventually, the RPN targets 205 have two components for each image: a vector which states if each of the nba anchors is positive, neutral or negative, and the second component which is represented by delta coordinates between ground truth boxes and positive anchors among the RPNtapi selected anchors to train the RPN 204. Only mGTi ground truth instances are kept per image to avoid training on images with too many objects to detect. This parameter is important for training on natural scene images composing the COCO dataset as they might contain an overwhelming number of overlapping objects. [0075] Dimensions of the targets for one image are [nba] and [RPNtapi, (dy, dx, log log (dh), log log (dw))], where dy and dx are the normalised distance of the coordinates centers between ground truth and anchor boxes, whereas log log (dh) and log log (dw) respectively deal with the logarithm delta between height and width. Finally, the RPN 204 is a FCN aiming at predicting these targets. [0076] The RPN 204 feeds into the Proposal Layer 206. The Proposal Layer 206 does not consist of a network, but a filtering block which only keeps relevant suggestions from the RPN 204. As already stated, the RPN 204 produces scores for each of the nbaanchors with the probability to be characterised as positive, neutral or negative and the Proposal Layer 206 begins by keeping the highest scores to select the best pNMSl anchors.)
15. generating the first output using the model for the anchor shape; determining the second output; (Zatpeyakin: [0021]-[0026], [0027] FIG. 1B shows examples of a captured image 160 and identification of a facial shape for the image, in accordance with an embodiment. FIG. 1B includes a bounding box 162 having an identified face 164, a cropped bounding box 166, a default shape 168 and a fitted shape 170 of the system environment illustrated in FIG. 1A. As shown in FIG. 1B, the default shape 168 has predefined facial anchor points around eyes, noses, mouth, and jaw lines. The default shape 168 is centered and scaled according to the cropped bounding box 166. The default shape does not account for the actual position and alignment of the face in the image. By applying the prediction model as described below, the fitted shape 170 is identified that has better positions of the facial anchor points aligned with the identified face in the cropped bounding box 166 than the adjusted default shape 172.)
15. and comparing the first output and the second output, wherein a difference between the first output and the second output is used to adjust the parameters of the model. (Zatepyakin:[0028]- n one embodiment, the face alignment module 146 uses a barycentric mesh-based shape for prediction. The barycentric mesh-based shape uses a barycentric coordinates system. The barycentric coordinate system is a coordinate system in which a position of a point within an element (e.g., a triangle, or tetrahedron) is represented by a linear combination of its vertices. For example, when the element is a triangle, points inside the triangle can be represented by a linear combination of three vertices of the triangle. The mesh-based shape may consist of multiple triangles covering all the predefined facial anchor points. Each facial anchor point can be represented by a linear combination of vertices in an associated triangle. [0029] FIG. 2 shows an example of barycentric mesh-based shapes, in accordance with an embodiment. As shown in FIG. 2, a barycentric mesh-based default shape 210 has multiple triangles. The triangles cover all the predefined facial anchor points as shown in dash lines. The barycentric mesh-based default shape 210 may be adjusted according to the cropped bounding box 166. The adjusted barycentric mesh-based default shape 220 may determine updated positions of predefined facial anchor points 230 using vertices of the associated triangles to correspond the predefined facial anchor points to the default shape applied to the cropped bounding box 166. When applying the prediction model, a barycentric mesh-based fitted shape 240 is generated to adjust the mesh to the face within the image and include updated triangles. Then, the barycentric mesh-based fitted shape 240 may determine updated positions of predefined facial anchor points 250 using vertices of associated update triangles. Machefar: [0074] Eventually, the RPN targets 205 have two components for each image: a vector which states if each of the nba anchors is positive, neutral or negative, and the second component which is represented by delta coordinates between ground truth boxes and positive anchors among the RPNtapi selected anchors to train the RPN 204. Only mGTi ground truth instances are kept per image to avoid training on images with too many objects to detect. This parameter is important for training on natural scene images composing the COCO dataset as they might contain an overwhelming number of overlapping objects. [0075] Dimensions of the targets for one image are [nba] and [RPNtapi, (dy, dx, log log (dh), log log (dw))], where dy and dx are the normalised distance of the coordinates centers between ground truth and anchor boxes, whereas log log (dh) and log log (dw) respectively deal with the logarithm delta between height and width. Finally, the RPN 204 is a FCN aiming at predicting these targets. [0076] The RPN 204 feeds into the Proposal Layer 206. The Proposal Layer 206 does not consist of a network, but a filtering block which only keeps relevant suggestions from the RPN 204. As already stated, the RPN 204 produces scores for each of the nbaanchors with the probability to be characterised as positive, neutral or negative and the Proposal Layer 206 begins by keeping the highest scores to select the best pNMSl anchors.)
Zatepyakin does not teach: 15. “wherein the second output is based on an overlap of the labeled shape and the anchor shape;” or
Chen does teach:
15. The method of claim 14, wherein training comprises: receiving the training image and the labeled shape; (Chen: [0004]-[0006], Step (1), acquiring a color image I contains lanes, including three component images I(0), I(1), and I(2) corresponding to red, green, and blue color components of I, respectively; performing the CLAHE algorithm to enhance the contrast of I and generate K enhanced images, where the kth enhanced image, k = 0, 1, ..., K - 1, is formed by using the cth channel image I(c) as the input, where c is the remainder of k divided by 3.)
15. generating the first output using the model for the anchor shape; (Chen: [0006]-[0007] Step (2), constructing the deep convolutional neural network, which consists of an input module, a spatial attention module, a feature extraction module, and a detection module, for lane detection, and stacking the three component images of the color image as well as the K enhanced images generated by the CLAHE algorithm in step (1) as a tensor including K + 3 channels to serve as the input to the deep convolutional neural network. [0027] Step (1), I is set as a to-be-processed color image, including three component images I(0), I(1), and I(2), corresponding to red, green, and blue, respectively, and the CLAHE is performed K times on I to enhance the contrast of an input image and generate K enhanced images, where the kth enhanced image, k = 0, 1, ..., K - 1, is formed by using the cth channel image I(c) as the input. In one embodiment of the present disclosure, K = 6, and c is equal to the remainder of k divided by 3. Steps of the algorithm are as follows. First, an image I(c) is processed by using a sliding window. The height and the width of the sliding window are Mb + kΔ and Nb + kΔ, respectively, where Mb, Nb, and Δ are preset constants, which may be Mb = 18, Nb = 24, and Δ = 4. Second, the histogram of a block image covered by the sliding window is calculated and denoted as H; and if any histogram bin Hi exceeds a specified limit h, it is clipped as Hi = h, and amplitude differences are accumulated according to the following formula:
PNG
media_image1.png
42
220
media_image1.png
Greyscale
; [0028]-[0031])
15. determining the second output, wherein the second output is based on an overlap of the labeled shape and the anchor shape; (Chen: [0027] Step (1), I is set as a to-be-processed color image, including three component images I(0), I(1), and I(2), corresponding to red, green, and blue, respectively, and the CLAHE is performed K times on I to enhance the contrast of an input image and generate K enhanced images, where the kth enhanced image, k = 0, 1, ..., K - 1, is formed by using the cth channel image I(c) as the input. In one embodiment of the present disclosure, K = 6, and c is equal to the remainder of k divided by 3. Steps of the algorithm are as follows. First, an image I(c) is processed by using a sliding window. The height and the width of the sliding window are Mb + kΔ and Nb + kΔ, respectively, where Mb, Nb, and Δ are preset constants, which may be Mb = 18, Nb = 24, and Δ = 4. Second, the histogram of a block image covered by the sliding window is calculated and denoted as H; and if any histogram bin Hi exceeds a specified limit h, it is clipped as Hi = h, and amplitude differences are accumulated according to the following formula:
PNG
media_image1.png
42
220
media_image1.png
Greyscale
; [0028]-[0031] Step (3), the deep convolutional neural network for lane detection includes an input module, a spatial attention module, a feature extraction module, and a detection module. According to the data flow of the input module during forward propagation, input data first passes through a convolutional layer with 64 7 × 7 kernels and a stride of 2, and then a batch normalization operation and a ReLU activation operation are performed. The final part of the input module is a max pooling layer with a 3 × 3 sampling kernel and with a stride of 2. [0032] Step (4), output x of the input module is an M1 × N1 × C feature map, where M1 and N1 denote the height and the width, respectively, and C denotes the number of channels of the feature map.)
15. and comparing the first output and the second output, wherein a difference between the first output and the second output is used to adjust the parameters of the model. (Chen: [0033] Step (5), elements in the spatial attention map are taken as weights. Values of all positions of each channel of the output feature map x of the input module are multiplied by weights of corresponding positions of the spatial attention map to form a feature map, and then is fed to the feature extraction module in the embodiment of the present disclosure.[0036] Step (8), output of the detection module is a set of detected marking blocks, and a lane model is determined by the Hough transform algorithm using center coordinates of all the blocks in the set as inputs. Specifically, the center coordinates of a detected marking block is (υ, ν), and a lane is written as a straight line expressed in the polar coordinate system: ρ=ucos θ + vsin θ)
It would have been obvious before the effective filing date of the claimed invention was made to one of ordinary skill in the art to modify the machine learning algorithm of the combination of Zatpeyakin and Machefar’s for extraction of facial features using set of adjustable mesh shapes and anchor content with the teachings of Chen for machine learning and feature extraction of using image enhancement and contrast limited adaptive histogram equalization. The determination of obviousness is predicated upon the following findings: One skilled in the art would have been motivated to modify the combination of Zatpeyakin and Machefar in order to improve the overall feature detection and machine learning algorithms to leverage a contrast adaptive histogram equalization algorithm to ensure enhanced accuracy and robustness. Furthermore, the prior art collectively includes each element claimed (though not all in the same reference), and one of ordinary skill in the art could have combined the elements in the manner explained above using known engineering design, interface and/or programming techniques, without changing a “fundamental” operating principle of the combination of Zatpeyakin and Machefar, while the teaching of Chen continues to perform the same function as originally taught prior to being combined, in order to produce the repeatable and predictable result of enhancing overall computational efficiency and accuracy. It is for at least the aforementioned reasons that the examiner has reached a conclusion of obviousness with respect to the claim in question.
Consider Claim 16.
The combination of Zatpeyakin, Machefar and Chen teaches:
16. The method of claim 15, wherein training comprises: generating first outputs using the model for anchor shapes in the plurality of anchor shapes; determining second outputs, wherein the second outputs are based on an overlap of the labeled shape and the respective anchor shapes; and comparing the respective first outputs and the respective second outputs, wherein a difference between the respective first outputs and the respective second outputs is used to adjust the parameters of the model for the anchor shapes. (Chen: [0033] Step (5), elements in the spatial attention map are taken as weights. Values of all positions of each channel of the output feature map x of the input module are multiplied by weights of corresponding positions of the spatial attention map to form a feature map, and then is fed to the feature extraction module in the embodiment of the present disclosure. [0034] Step (6), Stage 2, Stage 3, and Stage 4 convolutional layer groups of ResNet50 are taken as the feature extraction module, and the output of Stage 3 serves as the input to Stage 4 as well as the input to a convolutional layer consists of 5nB kernels of size 1 × 1 and with a stride of 1, where nB denotes a preset number of detection boxes for each anchor point, and the convolutional layer finally outputs a feature map denoted by F1. Output of Stage 4 passes through a convolutional layer consists of 5nB kernels of size 1 × 1 and with a stride of 1, and the generated feature map is up-sampled and then sums corresponding elements one by one with F1 to generate an M2 × N2 × 5nB feature map F. Zatepyakin: [0027] FIG. 1B shows examples of a captured image 160 and identification of a facial shape for the image, in accordance with an embodiment. FIG. 1B includes a bounding box 162 having an identified face 164, a cropped bounding box 166, a default shape 168 and a fitted shape 170 of the system environment illustrated in FIG. 1A. As shown in FIG. 1B, the default shape 168 has predefined facial anchor points around eyes, noses, mouth, and jaw lines. The default shape 168 is centered and scaled according to the cropped bounding box 166. The default shape does not account for the actual position and alignment of the face in the image. By applying the prediction model as described below, the fitted shape 170 is identified that has better positions of the facial anchor points aligned with the identified face in the cropped bounding box 166 than the adjusted default shape 172. [0028]- n one embodiment, the face alignment module 146 uses a barycentric mesh-based shape for prediction. The barycentric mesh-based shape uses a barycentric coordinates system. The barycentric coordinate system is a coordinate system in which a position of a point within an element (e.g., a triangle, or tetrahedron) is represented by a linear combination of its vertices. For example, when the element is a triangle, points inside the triangle can be represented by a linear combination of three vertices of the triangle. The mesh-based shape may consist of multiple triangles covering all the predefined facial anchor points. Each facial anchor point can be represented by a linear combination of vertices in an associated triangle. [0029] FIG. 2 shows an example of barycentric mesh-based shapes, in accordance with an embodiment. As shown in FIG. 2, a barycentric mesh-based default shape 210 has multiple triangles. The triangles cover all the predefined facial anchor points as shown in dash lines. The barycentric mesh-based default shape 210 may be adjusted according to the cropped bounding box 166. The adjusted barycentric mesh-based default shape 220 may determine updated positions of predefined facial anchor points 230 using vertices of the associated triangles to correspond the predefined facial anchor points to the default shape applied to the cropped bounding box 166. When applying the prediction model, a barycentric mesh-based fitted shape 240 is generated to adjust the mesh to the face within the image and include updated triangles. Then, the barycentric mesh-based fitted shape 240 may determine updated positions of predefined facial anchor points 250 using vertices of associated update triangles. )
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TAHMINA ANSARI whose telephone number is 571-270-3379. The examiner can normally be reached on IFP Flex - Monday through Friday 9 to 5.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, SUMATI LEFKOWITZ can be reached on 571-272-3638. The fax phone numbers for the organization where this application or proceeding is assigned are 571-273-8300 for regular communications and 571-273-8300 for After Final communications. TC 2600’s customer service number is 571-272-2600.
Any inquiry of a general nature or relating to the status of this application or proceeding should be directed to the receptionist whose telephone number is 571-272-2600.
2662
/TA/
/TAHMINA N ANSARI/Primary Examiner, Art Unit 2674
May 30, 2026
Prosecution Insights