DETAILED ACTION
This Final Office Action is responsive to the claims filed on December 3, 2025.
Claim 10 is rejected under 35 USC 112(a).
Claims 1 and 4-10 are rejected under 35 USC 112(b).
Claims 1 and 4-10 are rejected under 35 USC 101.
Claims 1 and 4-8 are rejected under 35 USC 103 over Wang in view of Patil, He, and Liu.
Claim 9 is rejected under 35 USC 103 over Wang in view of Patil, He, Liu, and Qiao2.
Claim 10 is rejected under 35 USC 103 over Wang in view of Patil, He, Liu, and Truong.
Response To Arguments/Amendments
Objections To The Specification and Drawings: The arguments and amendments appear to overcome the rejections, which are withdrawn.
35 USC 112(b): The arguments and amendments will be addressed in the order presented in the response:
Images and Screen: As discussed in the prior rejection, the antecedence is not clear with the images. The Applicant is advised to amend to clarify which images are input which are maintained, and their relationship with the recited “screen.” Also, the use of “screen” as in the claim is incorrect. A screen is a monitor, not a frame of an image, as it appears the Applicant intended. The Applicant’s asserted interpretation of the meaning the terms on the record is appreciated, but the claims will need to be amended in order to overcome the rejections. These issues make the claims indefinite. The rejections are maintained.
Loss Function: The Applicant’s arguments and amendments have been considered and are persuasive. This rejection is withdrawn. However, this amendment, along with the corresponding amendment to the specification appear to introduce new matter. See the 35 USC 112(a) rejection below.
The Keypoint-RCNN Algorithm Is Added With A Keypoint Branch On The Basis Of A Mask-RCNN: This appears to be an awkward translation, and, contrary to the Applicant’s naked assertion that the amendments to claim 1 clarified this, its metes and bounds were not further clarified by the amendment. Please revisit the rejection for the reasons asserted by the examiner that the Applicant can choose to attempt to rebut.
High-Quality: That one provides an indication of an improvement to something that would improve quality is insufficient to overcome that the term “high-quality” is relative and its metes and bounds undiscernible.
35 USC 101: The Applicant’s arguments and amendments have been considered but are not persuasive. The arguments will be treated in the order presented in the Office Action.
Deep Learning Is Not Simply A Mathematical Algorithm: The Applicant argues that machine learning should be distinguished from math. However, claim 1 explicitly recites “performing regression.,” which is a mathematical operation. This argument is not persuasive.
The Claims Allegedly Solve A Longstanding Problem, SO They Allegedly Do Not Recite An Abstract Idea?: The Applicant indicates that a long standing problem is solved by the claims, including the ability to accurately measure the body size and weight of a pig while reducing stimulation on the animals. The Applicant asserts this means that it fails at Step 2A, Prong One. However, that is not the test of Step 2A, Prong One. Step 2A, Prong One assesses whether an abstract idea is recited. As indicated below and in the prior rejection, the claims recite mental processes and evaluations performable in the mind or with the aid of pen, paper, and/or a calculator.
The Claims Allegedly Recite Additional Limitations That Integrate The Abstract Idea Into A Practical Application At Step 2A, Prong Two: The Applicant continues to extoll the benefits of the alleged invention as a whole and makes a bald statement that the claim recites additional limitations that integrate the abstract idea into a practical application. However, the Applicant has not identified a single additional limitation outside of the abstract idea that integrates the abstract idea into a practical application. The rejections clearly identify additional limitations and why those fail to confer eligibility at Step 2A, Prong Two, and the Applicant has not provided any rebuttal arguments or evidence to demonstrate otherwise. As it stands, the claim is similar to Example 47, Claim 2 of the subject matter eligibility examples in that the system merely takes in data for analysis and outputs an evaluation. The recited additional limitations are insufficient to confer eligibility at Step 2A, Prong Two and Step 2B. Accordingly, the rejections are maintained.
35 USC 103: The arguments from the response will be addressed in the order they were presented.
(1) Wang Is A General Review Article That Allegedly Fails To Provide Motivation To Combine Elements of Methods Therein: The Applicant argues that Wang does not disclose a complete technical solution for combining all of the elements of the claim because it is a review article that recites many approaches to tackling similar problems. This is not persuasive. If a reference teaches all the elements the office action asserts, and the elements are being used in the same way as in the claim, then there is motivation to combine different elements of the reference with substitutions of components being used for the same purpose. Contrary to the assertion of the Applicant, a review article is an excellent starting point for a person of ordinary skill in the art to map out a path for solving a problem. The Applicant’s specification (e.g., [0002]) makes it clear that the industry has attempted to address the problem of determining the size and weight of a pig by image inference, rather than a stressful and inefficient direct measurement. The Applicant’s assertion that elements of a review article cannot be reasonably combined with other references to teach other elements being used for their ordinary purposes is incorrect is tantamount to attacking the references individually. See MPEP 2145((IV).
(2) Human Pose Estimation Allegedly Differs From Pig Pose Estimation: The Applicant argues that the keypoints for humans differ significantly from those of a pig such that a person skilled in the art would not consider the human pose concepts in pig estimation. This is not persuasive. The skeletal points between humans and pigs are very similar, except that humans do not have a pronounced tail and are bipedal. The pose estimation concept is the same for both humans and pigs. The Applicant then proceeds to attack the Patil reference individually, which, as demonstrated, is invalid under MPEP 2145(IV).
(3) He Allegedly Does Not Suggest Using Resnext As A Backbone: The Applicant argues that He, while suggesting using Resnet, does not suggest using Resnext as a backbone. That is not convincing. At the time of filing, ResNet and ResNext were known substitutes as backbones for networks. The Applicant makes a statement that the “unified network design of the present application improves feature consistency and reduces computational redundancy” far exceeds that of linear models. However, the vast majority of machine learning algorithms are non-linear, made so at least by conventional activation functions. The Applicant’s statement is tantamount to saying machine learning models are better than basic linear regression, which may generally be the case. If it is true, this applies to all machine learning techniques whether they deploy a backbone or not and whether the backbone deployed is ResNet or ResNext. The Applicant cannot claim to have invented machine learning models, so the alleged benefit conferred is not conferred by the elements of the claim. Should the Applicant wish to assert that the structure of the machine learning algorithms of the claim confer specific advantages over the architectures presented in the art, it would make sense to provide evidence, such as a declaration to support the assertion. As it stands, the Applicant has not made an evidentiary demonstration that the specific features of the claims are an improvement over anything other than methods that are not machine learning, generally.
(4) The Applicant Alleges The Combination Lacks Motivation:
(A) Human Pose Estimation Allegedly Differs From Pig Pose Estimation, Part II: See “Human Pose Estimation Differs From Pig Pose Estimation,” above.
(B) Liu Allegedly Teaches A Cow, Not A Pig: The Applicant appears to draw a line in the sand between detection of pig features and detection of cow features. However, as demonstrated in the Wang review article, the persons of ordinary skill in the art consider these methods and approaches to be analogous for all livestock, including cows, pigs, and horses. They are all quadrupedal with heads and tails that are arranged with a very similar physical structure with points that are similarly detected.
The Applicant also alleges that Liu fails to teach a “tail root.” However, Liu teaches a “tailhead,” which is the root from which the tail extends, so Liu teaches a “tail root” by the ordinary meaning of the terms the Applicant used. If the Applicant wanted to define the “tail root” as something else, which does not really make sense and would be contrary to the ordinary meaning of the terms, they should have provided a drawing identifying these specific points on the pig or clearer descriptions of these elements in the specification.
(C) Qiao2 Is Allegedly Not About Weight Estimation: Qiao2 was not introduced to teach pig weight and size estimation. It was brought in to teach the conventional practice of storing data regarding an animal with an identifier for effective retrieval. This is a common practice in the machine learning arts and in the art of livestock tracking, an element of pig size and weight estimation. The Applicant is again attacking reference individually rather than the combination.
(D) Truong Demonstrates A Common Feature of Machine Learning Models But Not Pigs: The technologies addressed here include both livestock management and machine learning. The Truong reference demonstrates common loss functions applied to many machine learning algorithms, regardless of application, as an element of machine learning technology. The references need not be perfect matches but can be elements of analogous art with a motivation to combine, which was demonstrated in the prior office action.
(5) Slant Correction Step: The Applicant argues that a teaching of rotation in prior art references is not the same as a “slant correction.” This is facially false. The purpose of rotating elements in bounding boxes is to bring the elements in the box to a comparable pose with objects in other bounding boxes, or slant correction. Contrary to the assertion by the Applicant the rotation in Wang can solve detection problems caused by occlusion, slant, and lighting changes in pig images.
(6) Synergy: The Applicant makes a bald assertion that the specific features of the machine learning models solve specific technical problems in pig body size and weight estimation, but the Applicant has provided no evidence to lllustrate this. For example, the Applicant extolls the value of using ResNet over ResNext for this particular application. The Applicant has not provided any evidence (e.g., experimental results or a declaration from an inventor) that the particular claimed framework is an improvement over anything else and/or why it should it be so. This leaves the public in doubt that the Applicant has done any work to demonstrate that the claim features, which amount to a specific arrangement of generic and conventional machine learning architectural features generally known to be good for producing inferences, does so at a higher level than other potential arrangements of these conventional machine learning architectural features. What is it about each choice of architectural feature that makes it ideal for pig detection? What about these specific architectural features confers the alleged advantages the Applicant has endorsed as providing the desired end of better/more efficient pig estimation. The specification has not answered this question, and the Applicant has yet to clarify this on the record. Accordingly, a bald assertion of synergistic effects is insufficient to overcome the rejection, the references of which teach the features of the claims with more than sufficient motivation to combine the references. The rejections of original claims 1-3 are maintained for the rejection of claim 1 that substantially (but not identically) incorporated claims 2 and 3 by amendment in the last response.
Claim Rejections - 35 USC § 112(a)
Claim 10 is rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
Claim 10 was amended to recite, “wherein the weight estimation model comprises a loss function, and the loss function uses a root mean square error function.” However, the specification fails to provide support for this feature. The specification repeatedly asserts that the loss function is an element of the machine learning model. In order for the Applicant to rely on implied support for the art, the feature of the claim should be inherent. Loss functions are used in multiple ways, and some of them are not used to train machine learning models. Accordingly, the features of claim 10 are not supported by the specification and re new matter.
Claim Rejections - 35 USC § 112(b)
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-10 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
“images”
Claim 1 recites “obtaining images” in step S1; “keypoints of the pigs in the images,” “removing images,” ”retaining images,” and “to obtain images” in step S2; “obtain images” in step S3; and “inputting the images” in S4. The antecedence attached to the word “images” makes it unclear on which images the method is operating. For purposes of examination, each instance of the images will be interpreted to be the originally obtained images of the pig.
“in a/the screen”
Claim 1 recites “in a screen” and “in the screen,” which appears to be an awkward translation using non-idiomatic English. For purposes of examination, “in a screen” will be interpreted to mean within the frame of an image.
The Keypoint-RCNN Algorithm Is Added With A Keypoint Branch On The Basis Of A Mask-RCNN
Claim 1 recites, “the Keypoint-RCNN algorithm is added with a keypoint branch on the basis of a Mask-RCNN.” This language is unclear. The claim language sounds like a poor translation and does not mean anything. It appears that the claim language was intended to claim the model as illustrated in the Applicant’s FIG. 4. However, based on the current language, it is broad enough to state a generic Keypoint-RCNN (e.g., without an added Mask Branch), which includes a keypoint branch with keypoint head and mask, and its basis is a Mask-RCNN (because the Keypoint-RCNN is a variation of and derivation from the Mask-RCNN). For purposes of examination, the feature will be examined as if it is reciting a Keypoint-RCNN. If the Applicant wants to claim both the Mask branch and an attached Keypoint branch, as illustrated in the Applicant’s FIG. 4, there must be an explicit statement that both branches are attached within the same model. This may be problematic, however, because the Box Offsets and Class Scores for Keypoint-RCNN and Mask-RCNN differ in dimension and nature. This means that the fully connected layers that output the box offsets and class scores would have to be different, and the specification fails to enable and describe this and fails to illustrate this. (See the Patil et al. reference of record). That is, it is likely that the network illustrated in Applicant’s FIG. 4 is not enabled, described, or correctly illustrated by this specification and likely incorrectly depicts the flow of information that would be necessary to attach keypoint and mask branches to the same foundation that includes a common backbone, ROIAlign layer, and the Flatten layer.
High-Quality
Claim 1 recites, “high-quality regions of interest.” High quality is a relative term of degree, so a person of ordinary skill in the art would not be able to determine the metes and bounds of the claim.
Regardless of the asserted interpretation used during examination, appropriate correction by amendment of the indefinite elements is required. Dependent claims depending from rejected claims are rejected based on their dependency.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
NOTE: Prior to the analysis under 35 USC 101, it should be noted that the approach of the claims is well-known in the art. As demonstrated in the Wang reference of record, which is a review of the common methods for livestock size and weight estimation before the effective filing date of the claims, the claims represent generic computer activity and well-understood, routine, and conventional activity that uses the methods and models for their generic purposes. See Wang, Page 4, Right Column, Last Paragraph – Page 8, Right Column, First Paragraph, which illustrates the exact steps of the independent claim in different words. Wang also describes a number of different algorithms tried, including RCNNs (Wang, Page 10, Left Column, Last Paragraph), and well-known variants, such as Mask RCNN and Keypoint RCNN, which is a variant of Mask RCNN (See the Patil reference on record, pages 8-10 with associated images). Each of these use a pretrained model backbone, including the popular ResNext backbone of the claims.
The method steps S1-S3 merely preprocess the images for segmentation, orientation, and eliminating partial images that will be poor predictors of weight. This uses a Keypoint-RCNN, which is a common variant of Mask-RCNN that provides feature maps of different scales to determine keypoints.
Step S4 uses the preprocessed images in a weight estimation model that also relies on keypoint determination using, what appears to be, a hybrid network that outputs both Mask-RCNN features and Keypoint-RCNN features from Mask-RCNN and Keypoint-RCNN branches, respectively. Not in the independent claim is that the weight estimation is conducted by a further set of convolutional layers that could (though, it is not clear from the specification) receive the outputs of the Mask-RCNN and Keypoint-RCNN branches for predicting the weight of the pig. It is unclear from the specification, but the convolutional layers may, based on informed inference, include a further instance of ResNext layers as a foundation for the weight estimation that receives the keypoints from the Mask-RCNN and Keypoint-RCNN branches as input. Also, the keypoints output by the Mask-RCNN and Keypoint-RCNN branches include size features of the pig, including distances between features, such as ears and elbows. However, the output of the keypoint features themselves can be characterized as “body size data” without any further processing.
Further, these claims are analogous to Example 47 of the Patent Eligibility Examples (https://www.uspto.gov/sites/default/files/documents/2024-AI-SMEUpdateExamples47-49.pdf), Claim 2, in that they merely ingest data, process the data, and output resulting data, whether it be for training a model or using a model for inference. There are no additional limitations that integrate the abstract idea into a practical application at Step 2A, Prong 2, or combine with the other elements of the claim to provide significantly more that would confer an inventive concept at step 2B. This should be contrasted with Example 47, claim 3, which demonstrates additional limitations that confer eligibility.
Independent Claim 1
Step 1
Claim 1 is directed to a process.
Step 2A, Prong 1
Claim 1 recites an evaluation, which is a mental process, and a mathematical operation, which is a mathematical concept. Mental processes and mathematical concepts are abstract ideas.
Specifically, Claim 1 recites (claims in bold italics, paragraph references are to the Applicant’s specification):
A method for estimating a body size and weight of a pig based on deep learning, the method comprising the following steps: ([0067]-[0069] A convolutional neural network is used to determine the keypoints and weights, which is an evaluation, a mental process that can practically be performed in the mind or with the aid of pen, paper, and/or a calculator.. The keypoints are used to calculate body size (e.g., by drawing lines), which is a mathematical operation, a mathematical concept)
[…]
S2, detecting, by using a keypoint detection algorithm, keypoints of the pig in the images to obtain a keypoint detection result, and removing images that the pig is incomplete in a screen and retaining images that the pig is complete in the screen according to the keypoint detection result; ([0050]-[0058] Image segmentation is used to isolate pixels in the image representing the pig, and images without a complete pig are removed, which are evaluations, mental processes that can practically be performed in the mind or with the aid of pen, paper, and/or a calculator.)
S3, detecting whether the pig is slanted in the screen, and correcting the screen of the slanted pig to obtain images that the pig is complete and not slanted in the screen; and ([0059]-[0061] Keypoint-RCNN and Mask-RCNN include an ROIAlign layer that aligns the pigs in the images that remain after the partial pig images are removed. This is an evaluation, a mental process that can practically be performed in the mind or with the aid of pen, paper, and/or a calculator.)
S4, inputting the images into a weight estimation model and calculating body size data according to the keypoint detection result to obtain the weight and body size data of the pig, ([0066]-[0069] This is using a “weight model” for inference of weight and using the keypoints determined as a step in the segmentation as body size estimation points. These are evaluations, which are mental processes that can practically be performed in the mind or with the aid of pen, paper, and/or a calculator. Further, the determination of size of the pig based on the determined keypoints, as shown in paragraph [0067], is, in its broadest reasonable sense, a calculation, which is a mathematical concept.)
wherein the keypoint detection algorithm in the step S2 is built based on a Keypoint-Recurrent Convolutional Neural Network (Keypoint-RCNN) algorithm, the weight estimation model in the step S4 is built based on a ResNext-101 feature extraction network, and the Keypoint-RCNN algorithm is added with a keypoint branch on the basis of a Mask-RCNN, and a feature extraction network of the Keypoint-RCNN algorithm adopts the ResNext-101 feature extraction network; and the weight estimation model in the step S4 uses the ResNext-101 feature extraction network. ([0050]-[0051] and [0062]-[0066] These merely specify elements of the abstract idea, the means for conducting the aforementioned evaluations, including structure and implicit elements of the training.)
wherein in the step S2, by using an instance segmentation algorithm, the images are subjected to instance segmentation first before the keypoints are detected, and pixels belonging to the pig in the images are marked; the instance segmentation algorithm is built based on a Mask RCNN instance segmentation network; and an instance segmentation process comprises: (Mental Evaluation, Mental Process – This wherein clause merely qualifies the abstract idea as an element of the abstract idea.)
inputting the images into the ResNext-101 feature extraction network in the Mask RCNN instance segmentation network to obtain a feature map; (Mental Evaluation, Mental Process – Assessing images to determine a feature map is an evaluation that is practically performable in the mind or with aid of pen, paper, and/or a calculator. This step merely qualifies the abstract idea as an element of the abstract idea.)
setting a fixed number of regions of interest for each pixel position of the feature map, inputting the regions of interest into a region proposal network in the Mask RCNN instance segmentation network to perform binary classification to obtain a foreground and a background, and performing coordinate regression, so as to obtain high-quality regions of interest; (Mental Evaluation, Mental Process – Choosing a number of regions of interest is an evaluation that is practically performable in the mind or with aid of pen, paper, and/or a calculator. This step merely qualifies the abstract idea as an element of the abstract idea.)
performing ROIAlign operation on the obtained regions of interest, namely, first establishing a correspondence between pixels of the original images and the feature map and then establishing a correspondence between the feature map and fixed features; and (Mental Evaluation, Mental Process – Determining and processing alignment of elements in images is an evaluation that is practically performable in the mind or with aid of pen, paper, and/or a calculator. This step merely qualifies the abstract idea as an element of the abstract idea.)
classifying the regions of interest in a fully connected layer, generating detection boxes of detected objects in the regions of interest, and performing regression on the regions of interest to make the detection boxes gradually approach correct positions of the detected objects, and performing segmentation in a fully convolutional layer, to finally obtain a result of instance segmentation (Mental Evaluation, Mental Process – Absent the use of generic machine learning computing components, the classifying, generaing, regression, and segmentation steps are all practically performable in the mind or with aid of pen, paper, and/or a calculator. These steps merely qualify the abstract idea as elements of the abstract idea.)
wherein in the step S2, the keypoints are detected after the instance segmentation, and the keypoint detection algorithm is built based on the Keypoint-RCNN algorithm; the keypoint detection algorithm is configured to detect and mark the keypoints, position of each keypoint is modeled as a separate one-hot mask, each type of keypoint has a mask, and only one pixel for each keypoint is marked as the foreground; and (Mental Evaluation, Mental Process – These merely qualify step S2, which is practically performable in the mind or with aid of pen, paper, and/or a calculator. These features merely qualify the abstract idea as elements of the abstract idea.)
and the weight estimation model in the step S4 uses the ResNext-101 feature extraction network. (Mental Evaluation, Mental Process – These merely qualify step S4, which is practically performable in the mind or with aid of pen, paper, and/or a calculator, a mental process, abstract idea. These features merely qualify the abstract idea as elements of the abstract idea.)
Claim 1 recites a mental process and a mathematical concept, which are abstract ideas under MPEP 2106.04(a)(2)(III) and MPEP 2106.04(a)(2)(I).
Claim 1 recites an abstract idea.
Step 2A, Prong 2
Claim 1 does not recite any additional limitations that integrate the abstract idea into a practical application.
Claim 1 recites the following additional limitations:
S1, obtaining images of the pig;
Step S1 merely gather(s) existing information (images of pigs) for evaluation. Mere data gathering is insignificant extra solution activity under MPEP 2106.05(g). Under Mere Data Gathering, an analogous example is provided: “iv. Obtaining information about transactions using the Internet to verify credit card transactions, CyberSource v. Retail Decisions, Inc., 654 F.3d 1366, 1375, 99 USPQ2d 1690, 1694 (Fed. Cir. 2011).” Under MPEP 2106.05(g), receiving data for evaluation is not significant in meaningfully limiting the invention, and the receiving of the data is necessary to the evaluations and/or mathematical operations of the claim. Step S1 adds nothing more than insignificant extra solution activity, so it/they does/do not integrate the abstract idea into a practical application at Step 2A, Prong Two.
Also, should it be found that the model and training and/or inference methods thereof are not abstract ideas, the model and training and/or inference methods thereof are recited at a high level of generality, so the model and training methods thereof are also generic computing elements generically computing model parameters and fail to integrate the abstract idea into a practical application under MPEP 2106.05(f). Also should it be found otherwise, the specific models used for the applications of the claim merely limit the abstract idea to a particular field of machine learning and/or use of machine learning for image analysis and fail to integrate the abstract idea into a practical application under MPEP 2106.05(h). While the conventionality of elements is not considered at Step 2A, Prong 2, the evidence provided in the analysis of Step 2B below illustrates the generic nature of the computing elements under 2106.05(f) and the reasons why and extent to which the specific modeling elements merely limit the abstract idea to a particular technical environment under MPEP 2106.05(h).
the keypoints obtained by segmentation comprise: a left ear root point, a right ear root point, a left front elbow point, a right front elbow point, a left rear elbow point, a right rear elbow point, a spinal back point, and a tail root point.
Also, under MPEP 2106.05(h), the elements related to the specific inputs, outputs and intermediates, merely limit the claim to livestock parameter determinations and fail to integrate the abstract idea into a practical application under MPEP 2106.05(h).
The additional limitations of claim 1 fail to integrate the abstract idea into a practical application.
Claim 1 is directed to the abstract idea.
Step 2B
S1, obtaining images of the pig;
Step S1 is storing and retrieving information from memory and also indicative of sending or receiving data, so the step is analogous to the examples cited in MPEP 2106.05(d)(II)(i) representing well-understood, routine, and conventional functions: […] i. Receiving or transmitting data over a network, e.g., using the Internet to gather data […] iii. Electronic recordkeeping […] iv. Storing and retrieving information in memory.
Because the additional limitation of step S1 is insignificant extra-solution activity (as illustrated under Step 2A Prong 2) and well-understood, routine, and conventional, the step fail(s) to combine with the other elements of the claim to provide significantly more than the abstract idea that would render the combination an inventive concept, under MPEP 2106.05(d) and MPEP 2106.05(g), at step 2B.
the keypoints obtained by segmentation comprise: a left ear root point, a right ear root point, a left front elbow point, a right front elbow point, a left rear elbow point, a right rear elbow point, a spinal back point, and a tail root point.
Also, under MPEP 2106.05(h), the elements related to the specific inputs, outputs and intermediates, merely limit the claim to livestock parameter determinations and fail to provide significantly more than the abstract idea that would render the combination an inventive concept, under MPEP 2106.05(h).
Also should it be found that the model and training and/or inference methods thereof are not abstract ideas, the specific models used for the applications of the claim are well-understood, routine, and conventional (WURC).
The following is WURC evidence:
Using Mask-RCNN and Keypoint-RCNN for image segmentation and keypoint determination, and for determining keypoints for size and weight estimation, and incorporating of ROIAlign into Mask-RCNN and Keypoint-RCNN for alignment: See (1) Rath, FIG. 2 and associated description on page 7; (2) Wang et al., Page 9, Table 3, Rudenko and associated description on page 10, Right Column, Second Paragraph; (3) Di-Febbo et al., Abstract; (4) He et al., Abstract, Page 4, Network Architecture- showing Head/Mask architecture; (5) Holt, Abstract; (6) Hong et al., Abstract, Page 94, Table 1 and Page 96, Table 2; (7) Hui, Pages 2-12; (8) Jatesiktat et al. [0130], [0140]; (9) Naslavsky et al., [0065]; (10) Patil et al., Page 1, Pages 8-10 showing that Keypoint-RCNN is just a modification of the head/mask portions of the Mask-RCNN model; (11) Rock et al., [0042]; (12) Salau et al., Simple Summary and Abstract on Page 1.; (13) Sant’Ana et al., Page 10, Conclusion; (14) Srivastav, [0071], [0093];
Using ResNext as a backbone for Keypoint-RCNN and/or Mask-RCNN or using it as an element of an inference model to benefit from its pretraining See (1) He et al., Abstract, Page 4, Network Architecture, and Table 1 on page 5; (2) Azimi et al. Page 437, Table 2; (3) Hong et al., Abstract, Page 94, Table 1, Page 102, Table 6; (4) Phalak [0322] and [0326]
Also, should it be found that the model and training and/or inference methods thereof are not abstract ideas, the model and training and/or inference methods thereof are recited at a high level of generality, so the model and training methods thereof are also generic computing elements generically computing model parameters and fail to combine with the other elements of the claim to provide significantly more than the abstract idea that would render the combination an inventive concept under MPEP 2106.05(f). Also should it be found otherwise, the specific models used for the applications of the claim merely limit the abstract idea to a particular field of machine learning and/or use of machine learning for image analysis and fail to combine with the other elements of the claim to provide significantly more than the abstract idea that would render the combination an inventive concept under MPEP 2106.05(h). The WURC evidence provided in the analysis of Step 2B below illustrates the generic nature of the computing elements under 2106.05(f) and the reasons why and extent to which the specific modeling elements merely limit the abstract idea to a particular technical environment under MPEP 2106.05(h).
Also, under MPEP 2106.05(h), the elements related to the specific inputs, outputs and intermediates, merely limit the claim to livestock parameter determinations and fail to combine with the other elements of the claim to provide significantly more than the abstract idea that would render the combination an inventive concept under MPEP 2106.05(h).
The additional limitations of claim 1 fail to combine with the other elements of the claim to provide significantly more than the abstract idea that would render the combination an inventive concept under.
Claim 1 is ineligible.
Dependent Claims
The dependent claims fail to provide additional limitations that confer eligibility.
Claim 4
Claim 4 does not provide any features to claim 1 to distinguish from Example 47, Claim 2.
The features of claim 4 merely qualify the evaluations recited in the abstract idea of claim 1 by using a ResNext101 backbone and common softmax activation and are, therefore, elements of the abstract idea. Therefore, claim 4 fails to recite additional limitations that would confer eligibility at Step 2A, Prong 2 and Step 2B for at least the same reasons as the steps in claim 1.
Should it be found otherwise, these are merely generic computing methods and elements for Keypoint-RCNN and Mask-RCNN networks, with evidence of their generic nature evident from the WURC evidence provided for claim 1, so they fail to confer eligibility under MPEP 2106.05(f).
Further, these steps merely limit the abstract idea to a particular field of segmentation by machine learning and/or doing so for the purpose of livestock analysis, so they fail to confer eligibility under MPEP 2106.05(h).
Further, these steps are WURC for the reasons demonstrated with respect to claim 1 and based on the same evidence provided, so the steps fail to confer eligibility at Step 2B.
wherein in the step S4, the weight estimation model is built based on the ResNext-101 feature extraction network, and a softmax layer for modifying the ResNext-101 feature extraction network is a fully connected layer, with an output quantity of 1.
Specific Further WURC Evidence – This states basic features of the Mask-RCNN and Keypoint-RCNN structure where the output of the backbone (in this case, ResNext101) undergoes a softmax operation which is fully connected and the total of the output is 1. (1) See He et al., Figure 5 on page 2966 and Page 296, Right Column, Second To Last Paragraph.; (2) See Patil et al., Page 15, discussing the loss function in Keypoint-RCNN.; (3) Qiao3 et al., Page 5, Section 3.5.4; (4) See Hong et al. Table 2 on page 96, showing that ResNext101 is one of a few common choices for backbone elements.
Therefore, Claim 4 fails to recite any additional limitations that integrate the abstract idea into a practical application at Step 2A, Prong 2 or that combine with the other elements of the claim to provide significantly more than the abstract idea that would confer an inventive concept at Step 2B.
Claim 4 is ineligible.
Claim 5
Claim 5 does not provide any features to claim 1 to distinguish from Example 47, Claim 2.
The features of claim 5 merely describe evaluations representing the training of the weight estimation machine learning model. These are evaluations, which are mental processes, abstract ideas. These merge with the abstract idea of claims 1 and 4. Therefore, claim 5 fails to recite additional limitations that would confer eligibility at Step 2A, Prong 2 and Step 2B for at least the same reasons as the steps in claims 1 and 4.
Should it be found otherwise, these are merely generic computing methods and elements for Keypoint-RCNN and Mask-RCNN networks, with evidence of their generic nature evident from the WURC evidence provided for claim 1, so they fail to confer eligibility under MPEP 2106.05(f).
Further, these steps merely limit the abstract idea to a particular field of training a machine learning model and/or doing so for the purpose of livestock analysis, so they fail to confer eligibility under MPEP 2106.05(h).
Further, these steps are WURC for the reasons demonstrated with respect to claims 1 and 4 and based on the same evidence provided, so the steps fail to confer eligibility at Step 2B.
wherein the weight estimation model is subject to model training after being built; and a training process is as follows: preparing a training data set comprising a plurality of the images of the pig and the weight of the pig corresponding to each image, segmenting the pig of each image in the training data set, and binarizing the images to obtain binarized images of the pig and the weight of the pig corresponding to the images; and dividing the training data set into a training set, a test set and a validation set according to a ratio of 6:2:2, inputting the training set into the weight estimation model to perform model training to determine model parameters, then testing, by the test set, an estimation accuracy of the weight estimation model, and finally inputting the validation set into the weight estimation model to further adjust the model parameters, so as to obtain a trained weight estimation model.
The 60:20:20 ratio is one of the most common training/validation/test splits, and training, validation, and testing on segmented (“binarized”) images is also a very common method. (1) Wang et al., Page 10, Left column, Last Paragraph; (2) Nomura et al., Page A1107, Left Column, First Paragraph; (3) Bahad et al. Page 80, Fourth Paragraph; (4) Lever et al., Page 703, Right Column, Last Paragraph (“typical 60/20/20 split”); (5) Phung et al., Page 3, Under The Bullet Points); (6) Kumar, Page 4, FIG:;
Therefore, Claim 5 fails to recite any additional limitations that integrate the abstract idea into a practical application at Step 2A, Prong 2 or that combine with the other elements of the claim to provide significantly more than the abstract idea that would confer an inventive concept at Step 2B.
Claim 5 is ineligible.
Claim 6
Claim 6 does not provide any features to claims 1, 4, and 5 to distinguish from Example 47, Claim 2.
The features of claim 6 merely describe evaluations representing the step S4 of claim 1. These are evaluations, which are mental processes, abstract ideas. These merge with the abstract idea of claims 1, 4, and 5. Therefore, claim 6 fails to recite additional limitations that would confer eligibility at Step 2A, Prong 2 and Step 2B for at least the same reasons as the steps in claims 1, 4, and 5.
Should it be found otherwise, these are merely generic computing methods and elements for Keypoint-RCNN and Mask-RCNN networks for keypoint features to input into a weight estimation layer set, with evidence of their generic nature evident from the WURC evidence provided for claims 1, 4, and 5, so they fail to confer eligibility under MPEP 2106.05(f).
Further, these steps merely limit the abstract idea to a particular field of training a machine learning model and/or doing so for the purpose of livestock analysis, so they fail to confer eligibility under MPEP 2106.05(h).
Further, these steps are WURC for the reasons demonstrated with respect to claims 1,4, and 5 and based on the same evidence provided, so the steps fail to confer eligibility at Step 2B.
wherein estimating, by the trained weight estimation model, the weight by the images of the pig comprises: inputting the images of the pig with the weight to be estimated into the weight estimation model, extracting, by a convolutional layer, features of the images to obtain the image features, and inputting the image features into the fully connected layer to finally output the estimated weight.
Further Specific WURC Evidence – (1) Wang et al. Page 2, Introduction, discussing the target being body weight. Also Figure 1, the two rightmost images on page 3 show this. Also, Page 3, Right Column – Page 4, First Paragraph discussing the features of the claim. Pages 4-8 show the feature extraction portion and pages 8-11 show methods for determining the weight using ANNs (with fully connected layers) to yield the body weight of the livestock based on the features. Wang et al. is a review that provide several studies that use these methods. Also, see all of the references in Wang et al. in Tables 1-3. (2) Zhang et al., Abstact; (3) QIAO1 Qiao et al, Abstract, and Page 7, Table 3.
Therefore, Claim 6 fails to recite any additional limitations that integrate the abstract idea into a practical application at Step 2A, Prong 2 or that combine with the other elements of the claim to provide significantly more than the abstract idea that would confer an inventive concept at Step 2B.
Claim 6 is ineligible.
Claim 7
Claim 7 does not provide any features to claim 1 to distinguish from Example 47, Claim 2.
The features of claim 7 merely qualifies the evaluation of step S4 of claim 1. These are evaluations, which are mental processes, and mathematical concepts, abstract ideas. These merge with the abstract idea of claim 1. Therefore, claim 7 fails to recite additional limitations that would confer eligibility at Step 2A, Prong 2 and Step 2B for at least the same reasons as the steps in claim 1.
Further, these steps merely limit the abstract idea to a particular field of training a machine learning model and/or doing so for the purpose of livestock analysis, so they fail to confer eligibility under MPEP 2106.05(h).
Further, these steps are WURC for the reasons demonstrated with respect to claim 1 and based on the same evidence provided, so the steps fail to confer eligibility at Step 2B.
wherein in the step S4, the body size data comprises: a shoulder width, a hip width, and a body length, and the body size data is calculated according to a distance between the keypoints.
Further Specific WURC Evidence – See Wang et al., Page 8, Right Column, Last Paragraph, which explicitly details hip width and body length. Shoulder width is specified on Page 5, Left Column, First Full paragraph. These are also shown in Table 1 that extends from page 6 to page 7.; (2) QIAO1 Qiao et al., Page 1 Introduction; (3) Holt et al., Scientific Abstract
Therefore, Claim 7 fails to recite any additional limitations that integrate the abstract idea into a practical application at Step 2A, Prong 2 or that combine with the other elements of the claim to provide significantly more than the abstract idea that would confer an inventive concept at Step 2B.
Claim 7 is ineligible.
Claim 8
Claim 8 does not provide any features to claim 1 to distinguish from Example 47, Claim 2.
The features of claim 8 merely qualifies the evaluation of step S3 of claim 1. These are evaluations, which are mental processes, abstract ideas. These merge with the abstract idea of claim 1. Therefore, claim 8 fails to recite additional limitations that would confer eligibility at Step 2A, Prong 2 and Step 2B for at least the same reasons as the steps in claim 1.
Further, these steps merely limit the abstract idea to a particular field of training a machine learning model and/or doing so for the purpose of livestock analysis, so they fail to confer eligibility under MPEP 2106.05(h).
Further, these steps are WURC for the reasons demonstrated with respect to claim 1 and based on the same evidence provided, so the steps fail to confer eligibility at Step 2B.
wherein in the step S3, the correcting is to correct the slanted pig with a minimum circumscribed rectangle.
Specific Further WURC Evidence – (1) Wang discusses rotation, scaling, and coordinate transformations on Page 8, Right Column, Third Paragraph.; (2) Fernandes et al., page 499, Figure 2, rotates and centers the pig in the minimum size rectangle for consideration; (3) CCLAUS et al. explicitly states that the images are rotated and custom “smallest” bounding boxes on Page 9, Bounding Boxes… bounding boxes are generally generated to be the smallest that encompass all of the pixels segmented into the object of interest, e.g., a pig.
Therefore, Claim 8 fails to recite any additional limitations that integrate the abstract idea into a practical application at Step 2A, Prong 2 or that combine with the other elements of the claim to provide significantly more than the abstract idea that would confer an inventive concept at Step 2B.
Claim 8 is ineligible.
Claim 9
Claim 9 does not provide any features to claim 1 to distinguish from Example 47, Claim 2.
The features of claim 9, in their broadest sense merely recite that the body size and weight data of the pig is associated with an identifier that is also associated with feature data representing “back features” that can be used to identify the pig. This does not claim any determination but merely discusses how data is stored. Accordingly, this is insignificant extra-solution activity under MPEP 2106.05(g) similar to the recited examples: “iv. Obtaining information about transactions using the Internet to verify credit card transactions,” “v. Consulting and updating an activity log,” “i. Limiting a database index to XML tags,” “iii. Selecting information, based on types of information and availability of information in a power-grid environment, for collection, analysis and display,” “iii. Selecting information, based on types of information and availability of information in a power-grid environment, for collection, analysis and display,” Because the features of claim 9 are insignificant extra-solution activity, they cannot integrate the abstract idea into a practical application at Step 2A, Prong 2.
Further, storing, indexing, and retrieving data is WURC similar to the following examples under MPEP 2106.05(d): “i. Receiving or transmitting data over a network,” “iii. Electronic recordkeeping,“ “iv. Storing and retrieving information in memory” “v. Electronically scanning or extracting data from a physical document,” “vi. Arranging a hierarchy of groups, sorting information, eliminating less restrictive pricing information and determining the price,”
Because the features of claim 9 are WURC and insignificant extra-solution activity, they do not combine with the other elements of the claim to provide significantly more than the abstract idea that would confer an inventive concept under MPEP 2106.05(d) and 2106.05(g) at Step 2B
Claim 9 is ineligible.
Claim 10
Claim 10 does not provide any features to claims 1 and 4 to distinguish from Example 47, Claim 2.
These features merely qualify the training of the model by specifying a loss function, which is an evaluation, a mental process, an abstract idea. This merges with the abstract idea of claim 4, so it does not provide any additional limitations to integrate the abstract idea into a practical application at Step 2A, Prong 2 or to combine with other elements of the claim to provide significantly more than the abstract idea that would confer an inventive concept at step 2B.
If it should be found that the model and training methods are not elements of the abstract idea, the model and training methods are generic computing elements that do not confer eligibility under MPEP 2106.05(f).
Further, these steps merely limit the abstract idea to a particular field of training a machine learning model and/or doing so for the purpose of livestock analysis, so they fail to confer eligibility under MPEP 2106.05(h).
Also, the use of the RMSE as a loss function in ML model training is WURC
wherein the weight estimation model comprises a loss function, and the loss function uses a root mean square error function.
Further Specific Evidence of WURC – See (1) Wang et al., Page 9, Table 3, Page 11, Left Column, Last Paragraph;(2) Sant’Ana, Page 7, Table 3; (3) Qiao1 et al., Page 7, Table 3; (4) Truong Page 4, So, what…? Page 5 “Importantly, the way that we evaluate classification and regression predictions varies and does not overlap, for example: - Classification predictions can be evaluated using accuracy, whereas regression predictions cannot. - Regression predictions can be evaluated using root mean squared error, whereas classification predictions cannot.” Page 7 “Cross-Entropy Loss (or Log Loss) It measures the performance of a classification model whose output is a probability value between 0 and 1. Cross-entropy loss increases as the predicted probability diverges from the actual label.”) (Additionally, the substitution of the cross-entropy loss function for the root mean square error loss function would result in a combination of prior art elements according to known methods to yield predictable results, would be obvious to try among a finite list of common (often with premade functions in programming languages) loss functions, and would be an obvious substitute, all based on the demonstrated evidence and explanation.
Therefore, Claim 10 fails to recite any additional limitations that integrate the abstract idea into a practical application at Step 2A, Prong 2 or that combine with the other elements of the claim to provide significantly more than the abstract idea that would confer an inventive concept at Step 2B.
Claim 10 is ineligible.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1 and 4-8: Wang in view of Patil, He, and Liu
Claim(s) 1 and 4-8 are rejected under 35 U.S.C. 103 as being unpatentable over NPL: “ASAS-NANP SYMPOSIUM: Applications of machine learning for livestock body weight prediction from digital images” by Wang et al. (Wang) in view of NPL “Human Pose Estimation using Keypoint RCNN in PyTorch” by Patil et al. (Patil), NPL: “Mask R-CNN” by He et al. (He), and NPL: “Video analytic system for detecting cow structure” by Liu et al. (Liu).
Claim 1
Regarding claim 1, Wang teaches:
A method for estimating a body size and weight of a pig based on deep learning, the method comprising the following steps: (Wang Abstract “With the concomitant advancement of contactless electro-optical sensors (e.g., 2D, 3D, infrared cameras), computer vision (CV) technologies, and artificial intelligence fields such as machine learning (ML) and deep learning (DL), 2D and 3D images have started to be used as biometric and morphometric proxies for BW estimations. This manuscript provides a review of CV-based and ML/DL-based BW prediction methods and discusses their strengths, weaknesses, and industry applicability potential.” Page 3, Right Column, Last Paragraph – Page 4, Left Column, First Paragraph “The remaining of this review is structured as follows. First typical biometric and morphometric measurements used for livestock BW estimation is provided. The next section discusses CV methods that are typically used for digital image processing and feature extraction, whereas the latter section will review” -First, measurements of body features for determining body size are extracted from the images. Then, the body features are used as input to an ML model to determine body weight
S1, obtaining images of the pig; (Wang Page 4, Left Column, Last Paragraph – Right Column, First Paragraph “Different types of 2D and 3D optical sensors have been successfully used mostly for morphometric measurements and in a limited way for biometric identification of animal behaviors (Porto et al., 2013; Banhazi and Tscharke, 2016; Nasirahmadi et al., 2017; Li et al., 2019). The 2D sensors include regular 2D digital cameras, thermal cameras (Stajnko et al., 2008), and systems of cameras capable to extrapolate 3D models from a series of 2D images.” Also, see Table 1 presented on pages 5-7. – Images of pigs are obtained.)
S2, detecting, by using a keypoint detection algorithm, keypoints of the pig in the images to obtain a keypoint detection result, and removing images that the pig is incomplete in a screen and retaining images that the pig is complete in the screen according to the keypoint detection result; (Wang Page 4, Right Column, Last Paragraph “The detection stage typically relies on weaker or stronger assumptions related to what is expected to be in front of the cameras, depending on the position of the cameras. […] More advanced technologies include face detection (Yao et al., 2019), muzzle detection (Noviyanto and Arymurthy, 2013; Tharwat et al., 2014), automatic detection via ML, and DL approaches applied to surveillance videos (Zhang et al., 2018a) or hybrid systems that combine radio-frequency identification sensors and CV technologies (Velez et al., 2013). A review of cattle detection methods is presented by Awad (2016). When 3D cameras are used, the distance between the sensor and the subject (depth) is proportional to the intensity of the pixels in the image and serves as a great classifier for foreground or background image components.” Page 8, First Paragraph – “A postprocessing step consisting of a number of quality control criteria is needed to select valid frames with correctly positioned animals that are completely included in the frame, not touching boundary walls and having a straight posture. Failing to control the quality of an image can lead to inaccurate image-based biometric morphometric measurements in the feature extraction stage.” Also, see Table 1 presented on pages 5-7. – Keypoint detection is conducted and images that capture only parts of animals are removed.)
S3, detecting whether the pig is slanted in the screen, and correcting the screen of the slanted pig to obtain images that the pig is complete and not slanted in the screen; and (Wang Page 8, Left Column, Step 2 “Automatic methods are typically focused on specific body parts identification and span a wide complexity spectrum, from simple background subtraction approaches (Yang and Teng, 2007), edge detection (Senthilkumaran and Rajesh, 2009) via operators such as Prewitt (Prewitt, 1970), Canny (Canny, 1986), Sobel, Laplacian (Kimmel and Bruckstein, 2003), and masking (e.g. Kirsch, Robinson) to complex operations for shape recognition such as the use of Hough transform (Hough, 1962) for identification of round objects (Fernandes et al., 2019) and super-pixel methods such as the Simple Linear Iterative Clustering (Achanta et al., 2012). Postprocessing steps could include black and white or color-based masking, rotation, scaling and coordinate transformations (e.g., conversions to polar coordinates).” – Images are rotated to not be slanted in the frame of the image.)
S4, inputting the images into a weight estimation model and calculating body size data according to the keypoint detection result to obtain the weight and body size data of the pig, (Wang Page 8, Step 3 “The third stage is focused on morphometric and biometric measurements extraction and can be grouped into two categories as suggested by Fernandes et al. (2019): body measurements and shape descriptors. The body measurements include lengths, widths, areas, and volumes and can be calculated either manually as described in the segmentation stage or automatically via superposing a grid on top of the segmented animal and extracting equidistant measurements along a line, curve, contour, or an area. The volumetric measurements are rough approximations given the limited camera views of either the top or the side of an animal. The shape descriptors reported in the literature include generic ones such as Fourier descriptors (Fernandes et al., 2019) and fast point feature histograms (Huang et al., 2019). […] “A summary of CV methods for processing images used in the prediction of BW in four livestock species (cattle, pigs, sheep, goats) is presented in Table 2. Moreover, a detailed species-specific review of CV methods for pigs’ BW estimations is available in the literature (Li et al., 2014).” Page 10, Last Paragraph – Page 11, First Paragraph “The features extracted were: 1) body measurements, including apparent volume, surface area, length, height and width, and eccentricity; 2) 360 equidistant measurements of the polar shape contour of the top view image; and 3) the corresponding 360 Fourier descriptor features of the same polar shape contour. The body measurements were extracted from the 3D images and converted to metric scale values using the intrinsic focal length of the Kinect depth camera. The pig volume was calculated as the sum of pixels’ volumes.” Also, See Tables 2 and 3) – The determined keypoints are used to determine body size. Page 8, Right Column, First Paragraph-Second Paragraph “A summary of CV methods for processing images used in the prediction of BW in four livestock species (cattle, pigs, sheep, goats) is presented in Table 2. Moreover, a detailed species-specific review of CV methods for pigs’ BW estimations is available in the literature (Li et al., 2014). […] While predicting BW of farm animals from biometric and morphometric measurements observed at different growth periods in cattle, pigs, sheep, and goats has been the focus of many past research studies, which applied traditional statistical regression techniques such as linear, multiple, and ridge regression, their success was limited by the multicollinearity and complex relationships among measurements (variables). To capture and explain such complex inter-variable relationships, a limited number of recent studies have reported the successful application of various ML and DL methods for predicting BW using features extracted from 2D and 3D digital images (Table 3).” – The determined keypoints are input into a weight determination ML model to predict a weight of a pig or other livestock.)
wherein the keypoint detection algorithm in the step S2 is built based on [Mask-RCNN] algorithm, the weight estimation model in the step S4 is built based on [a Mask-RCNN algorithm]. (Wang Page 10, Right Column, Second Paragraph “Rudenko et al. (2020) applied ANNs and CV to identify the cow’s breed and estimate their BW. Cow images taken at different angles by synchronized cameras were fed to a Mask RCNN to determine the breed and position of each subject. Then, withers height, hip height, body length, and width of a cow were determined using the stereopsis method from 3D images acquired with an Intel RealSense D435i camera using the position of the cow detected by Mask RCNN in the previous step. Finally, the obtained data about the species and its size were fed to a multilayer perceptron (MLP) to estimate the live weight of the animals.” – Mask-RCNN networks are used to extract keypoints to determine body size by calculation and determine body weight using a multilayer perceptron, which is a fully connected ANN to which convolution is typically applied for training and inference.)
wherein in the step S2, by using an instance segmentation algorithm, the images are subjected to instance segmentation first before the keypoints are detected, and pixels belonging to the pig in the images are marked; and an instance segmentation process comprises: (Wang Page 8, Left Column, Step 2 and Step 3 “The segmentation stage can include a large number of methods that can be classified as manual and automatic. […] The third stage is focused on morphometric and biometric measurements extraction and can be grouped into two categories as suggested by Fernandes et al. (2019): body measurements and shape descriptors. The body measurements include lengths, widths, areas, and volumes and can be calculated either manually as described in the segmentation stage or automatically via superposing a grid on top of the segmented animal and extracting equidistant measurements along a line, curve, contour, or an area. The volumetric measurements are rough approximations given the limited camera views of either the top or the side of an animal. The shape descriptors reported in the literature include generic ones such as Fourier descriptors (Fernandes et al., 2019) and fast point feature histograms (Huang et al., 2019). – Segmentation is performed before the keypoint detection. This is a standard element of the Mask-RCNN and Keypoint-RCNN models.;
Wang teaches the use of a Mask-RCNN to determine keypoints for body size and weight estimation. The claimed Keypoint-RCNN is a minor variant of the Mask-RCNN with a different head and mask portion that provides differently defined features as keypoints. Wang fails to explicitly teach, but Wang in view of Patil teaches:
wherein the keypoint detection algorithm in the step S2 is built based on Keypoint-Recurrent Convolutional Neural Network (Keypoint-RCNN) algorithm, the weight estimation model in the step S4 is built based on and the Keypoint-RCNN algorithm is added with a keypoint branch on the basis of a Mask-RCNN,. (Patil Pages 7-11 demonstrate the evolution from Faster RCNN to Mask-RCNN to the most evolved Keypoint-RCNN. Pages 10-11 Shows an image with a Keypoint RCNN that includes on the top-left a Keypoint-CRNN head and mask, representing a Keypoint-RCNN branch. – This illustrates that Keypoint-RCNN is an evolved and accepted version of (and therefore is “added with a keypoint branch on the basis of a) Mask-RCNN. Page 9 also shows an image of basic Mask-RCNN with a Mask-RCNN head and mask in a mask branch. – This illustrates the similarity between and the substitutability of Keypoint-CNN and Mask-RCNN. Page 14, Code Snippet In The Middle of the Page, Line 2 - Patil also demonstrates that there is a premade function for Keypoint-RCNN in PyTorch, a popular ML software, showing that the Keypoint-RCNN is an obvious and well-known variant of Mask-RCNN. See the figures that follow from Patil pages 8-11)
PNG
media_image1.png
1024
576
media_image1.png
Greyscale
PNG
media_image2.png
1024
576
media_image2.png
Greyscale
wherein in the step S2, by using an instance segmentation algorithm, the images are subjected to instance segmentation first before the keypoints are detected, and pixels belonging to the pig in the images are marked; the instance segmentation algorithm is built based on a Mask RCNN instance segmentation network; and an instance segmentation process comprises: (Patil Pages 8-10 show the structures of the Keypoint and Mask CRNNs and their ordered layers.)
inputting the images into the ResNext-101 feature extraction network in the Mask RCNN instance segmentation network to obtain a feature map; (Patil Page 10 shows the input images being input into the feature extractor layers of the Backbone, which include the He ResNext-101 Backbone demonstrated with respect to claim 1. See the below image.)
PNG
media_image3.png
111
345
media_image3.png
Greyscale
setting a fixed number of regions of interest for each pixel position of the feature map, inputting the regions of interest into a region proposal network in the Mask RCNN instance segmentation network to perform binary classification to obtain a foreground and a background, and performing coordinate regression, so as to obtain high-quality regions of interest; (Patil Page 10, Keypoint-RCNN image – Binarization is conducted to determine background and foreground to make a number (N) of regions of interest of a quality consistent with the level of coordinate regression.)
PNG
media_image4.png
110
379
media_image4.png
Greyscale
performing ROIAlign operation on the obtained regions of interest, namely, first establishing a correspondence between pixels of the original images and the feature map and then establishing a correspondence between the feature map and fixed features; and (Patil Page 10, Keypoint-RCNN image – The feature maps are fed into the ROI Align layer, which establishes a correspondence between pixels of the original images and the feature map and establishes a correspondence between the feature map and fixed features)
PNG
media_image5.png
199
410
media_image5.png
Greyscale
classifying the regions of interest in a fully connected layer, generating detection boxes of detected objects in the regions of interest, and performing regression on the regions of interest to make the detection boxes gradually approach correct positions of the detected objects, and performing segmentation in a fully convolutional layer, to finally obtain a result of instance segmentation. (Patil Page 10, Keypoint-RCNN image – The box offsets (location of boxes) of thee ROIs are generated from fully connected layers, the regression on the ROIs form the bounding boxes and there is segmentation in a convolutional layer, which provides a segmented image.)
PNG
media_image6.png
383
484
media_image6.png
Greyscale
wherein in the step S2, the keypoints are detected after the instance segmentation, and the keypoint detection algorithm is built based on the Keypoint-RCNN algorithm; the keypoint detection algorithm is configured to detect and mark the keypoints, position of each keypoint is modeled as a separate one-hot mask, each type of keypoint has a mask, and only one pixel for each keypoint is marked as the foreground; and (Patil Pages 11-12 show that image segmentation is performed first on page 11, and then keypoints are generated on page 12. Page 10 “Keypoint RCNN slightly modifies the existing Mask RCNN, by one-hot encoding a keypoint (instead of the whole mask) of the detected object.” – Each keypoint is one-hot encoded as its own mask. The point is a single pixel. This is illustrated in the images on page 12.)
PNG
media_image7.png
421
952
media_image7.png
Greyscale
the keypoints obtained by segmentation comprise: a left ear root point, a right ear root point, a left front elbow point, a right front elbow point, (Patil Page 18, shows many of these points on a human including a left ear root point, a right ear root point, a left front elbow point and a right front elbow point.)
PNG
media_image8.png
496
665
media_image8.png
Greyscale
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claims to modify the Mask-RCNN model and training and inference methods thereof of Wang with the Keypoint-RCNN model and training and inference methods thereof of Patil because the person of ordinary skill in the art would be motivated, based on the use of the Mask-RCNN in and the expressed desire to explore the strengths of the methods taught in Wang to use the more evolved iteration of the Mask-RCNN model and training and inference methods thereof, the Keypoint-RCNN model and training and inference methods thereof of Patil, with diverse applications. (Wang Abstract “With the concomitant advancement of contactless electro-optical sensors (e.g., 2D, 3D, infrared cameras), computer vision (CV) technologies, and artificial intelligence fields such as machine learning (ML) and deep learning (DL), 2D and 3D images have started to be used as biometric and morphometric proxies for BW estimations. This manuscript provides a review of CV-based and ML/DL-based BW prediction methods and discusses their strengths, weaknesses, and industry applicability potential.” Page 10, Right Column, Second Paragraph “Rudenko et al. (2020) applied ANNs and CV to identify the cow’s breed and estimate their BW. Cow images taken at different angles by synchronized cameras were fed to a Mask RCNN to determine the breed and position of each subject. Then, withers height, hip height, body length, and width of a cow were determined using the stereopsis method from 3D images acquired with an Intel RealSense D435i camera using the position of the cow detected by Mask RCNN in the previous step. Finally, the obtained data about the species and its size were fed to a multilayer perceptron (MLP) to estimate the live weight of the animals.”; Patil Page 7, Evolution of Keypoint RCNN Architecture “You have seen how Keypoint RCNN followed Mask RCNN, which in turn came after Faster RCNN. So, when introducing Keypoint RCNN, a brief overview of its predecessors is important.” Pages 23-24, Conclusion “We set out to explore Keypoint-Detection, using a variant of Mask RCNN to detect joints in a human body. Only after a brief overview of its predecessors did we go into the nitty-gritties of Keypoint-RCNN, and study its diverse applications.”). Further, it would have been an obvious and simple substitution to substitute the Keypoint-RCNN of Patil for the Mask-RCNN of Wang, whose functions were known in the art, and between which a person of ordinary skill in the art could have substituted to achieve predictable results (e.g., keypoint generation), under MPEP 2141(III)(B).
Wang in view of Patil teach Mask-RCNN and Keypoint-RCNN networks with pretrained feature extractor (e.g., the ResNet50 network of the premade PyTorch model in Patil Page 14, Code Snippet In The Middle of the Page, Line 2), but Wang in view of Patil fail to explicitly teach, but Wang in view of Patil and He teaches:
wherein the keypoint detection algorithm in the step S2 is built based on a ResNext-101 feature extraction network, and the Keypoint-RCNN algorithm is added with a keypoint branch on the basis of a Mask-RCNN, and a feature extraction network of the Keypoint-RCNN algorithm adopts the ResNext-101 feature extraction network; and the weight estimation model in the step S4 uses the ResNext-101 feature extraction network. (He Page 4, Left Column, Second Paragraph “We denote the backbone architecture using the nomenclature network-depth-features. We evaluate ResNet [15] and ResNeXt [35] networks of depth 50 or 101 layers. The original implementation of Faster R-CNN with ResNets [15] extracted features from the final convolutional layer of the 4-th stage, which we call C4. This backbone with ResNet-50, for example, is denoted by ResNet-50-C4. This is a common choice used in [15, 7, 17, 31].” Also See Page 6, Table 5(a) – He teaches using ResNext-101 as a backbone/basis for Mask-RCNN and, therefore, any direct derivatives thereof, as the Keypoint-RCNN of Patil.)
PNG
media_image9.png
329
445
media_image9.png
Greyscale
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claims to modify the Mask-RCNN and/or Keypoint-RCNN of Patil with the Resnext101 backbone of He because a person of ordinary skill in the art would be motivated, based on the desire expressed desire in Patil to use the most evolve models and explore the diverse applications of Keypoint-RCNN, to modify the Keypoint-RCNN mode in Patil that uses a Resnet50 backbone for the ResNext101 backbone of He that is better than Resnet.
(Patil Page 7, Evolution of Keypoint RCNN Architecture “You have seen how Keypoint RCNN followed Mask RCNN, which in turn came after Faster RCNN. So, when introducing Keypoint RCNN, a brief overview of its predecessors is important.” Pages 23-24, Conclusion “We set out to explore Keypoint-Detection, using a variant of Mask RCNN to detect joints in a human body. Only after a brief overview of its
predecessors did we go into the nitty-gritties of Keypoint-RCNN, and study its diverse applications.”; He Page 6, Table 5(a) “Backbone Architecture: Better backbones bring expected gains: deeper networks do better, FPN outperforms C4 features, and ResNeXt improves on ResNet.). Further, it would have been an obvious and simple substitution to substitute the ResNext101 of He for the ResNet50 of Patil, whose functions were known in the art, and between which a person of ordinary skill in the art could have substituted to achieve predictable results (e.g., keypoint generation), under MPEP 2141(III)(B).)
Wang in view of Patil and He do not appear to explicitly teach, but Wang in view of Patil, He, and Liu teaches:
the keypoints obtained by segmentation comprise: a left ear root point, a right ear root point, a left front elbow point, a right front elbow point, a left rear elbow point, a right rear elbow point, a spinal back point, and a tail root point. (Liu Page 2, Fig. 1 (shown below) which shows these points from the rear perspective, and the associated description “Fig. 1. The proposed cow structural model. 4 blue head region points: A:nose, B:head, C:top of neck, J:bottom of neck. 5 red body region points, D:shoulder, E:spine, G:tailhead, H:mid-thigh, I:bottom of shoulder. 8 white leg and hoof points, with name format: Right/Left + Front/Back + Leg/Hoof. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.”)
PNG
media_image10.png
410
617
media_image10.png
Greyscale
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claims to modify the livestock measurements presented (e.g., in Table 1) in Wang by the keypoints from which to derive measurements of Liu because the person of ordinary skill in the art would be motivated, based on the expressed purpose in Wang to use the keypoints taken to determine livestock body size measurements (e.g., for input into an ML model to determine a body weight), to look to Liu which provides better detection results and successful isolation of body keypoints of livestock. (Wang Abstract “With the concomitant advancement of contactless electro-optical sensors (e.g., 2D, 3D, infrared cameras), computer vision (CV) technologies, and artificial intelligence fields such as machine learning (ML) and deep learning (DL), 2D and 3D images have started to be used as biometric and morphometric proxies for BW estimations.” Page 3, Last Paragraph – Page 4, First Paragraph “The next section discusses CV methods that are typically used for digital image processing and feature extraction, whereas the latter section will review work related to ML methods applied to the BW prediction problem. The Conclusions section summarizes the reviewed work and identifies new avenues of research in the area.”; Liu Abstract “side-view cow structural model is designed to describe the spatial positions of the joints (keypoints) of the cow, and we develop a system using deep learning to automatically extract the structural model from videos. The proposed system can detect multiple cows in the same frame and provides robust performance for the body region under practical challenges like obstacles (fences) and poor illumination. Compared to other object detection methods, this system provides better detection results and successfully isolates the body keypoints of each cow even when the cows are close to each other.”)
Claim 4
Regarding claim 4, Wang in view of Patil, He, and Liu teaches the features of claim 1 and further teaches:
wherein in the step S4, the weight estimation model is built based on the ResNext-101 feature extraction network, and a softmax layer for modifying the ResNext-101 feature extraction network is a fully connected layer, with an output quantity of 1. (Patil Page 15, Loss Function in Keypoint-RCNN “As in Keypoint Detection, each Ground-Truth keypoint is one-hot-encoded, across all the K channels, in the featuremap of size [K=17, 56, 56], fora single object. For each visible Ground-Truth, channel wise Softmax (instead of sigmoid), from the final featuremap [17, 56, 56]” – The output goes to a softmax, which is a fully connected layer that outputs a probability for each input to the layer, the total probabilities of the outputs sum to 1.)
Claim 5
Regarding claim 5, Wang in view of Patil, He, and Liu teaches the features of claim 4 and further teaches:
wherein the weight estimation model is subject to model training after being built; and a training process is as follows: preparing a training data set comprising a plurality of the images of the pig and the weight of the pig corresponding to each image, segmenting the pig of each image in the training data set, and binarizing the images to obtain binarized images of the pig and the weight of the pig corresponding to the images; and dividing the training data set into a training set, a test set and a validation set according to a ratio of 6:2:2, inputting the training set into the weight estimation model to perform model training to determine model parameters, then testing, by the test set, an estimation accuracy of the weight estimation model, and finally inputting the validation set into the weight estimation model to further adjust the model parameters, so as to obtain a trained weight estimation model. (Wang Page 10 “Two cameras were fixed in the structure of the water trough to collect the dorsal image of the animal when drinking and two other cameras were installed in the trough cover structure to acquire the profile images of the animals going into the trough. The data set was divided into training (60%), validation (20%), and testing (20%) sets. The authors reported that CNNs achieved the highest performance with a top model mean average error (MAE) of 23.19 ± 1.46 kg, which was nearly half the error of the top LR models proposed by de Moraes Weber et al. (2020) with an MAE value of 38.46 kg. – The training of the model is conducted using training, testing, and validation sets in one of the most common ratios of 6:2:2 for training, testing, and validation).
Claim 6
Regarding claim 6, Wang in view of Patil, He, and Liu teaches the features of claim 5 and further teaches:
wherein estimating, by the trained weight estimation model, the weight by the images of the pig comprises: inputting the images of the pig with the weight to be estimated into the weight estimation model, extracting, by a convolutional layer, features of the images to obtain the image features, and inputting the image features into the fully connected layer to finally output the estimated weight. (Wang Page 8, Right Column, ML and CV Methods for BW Prediction “While predicting BW of farm animals from biometric and morphometric measurements observed at different growth periods in cattle, pigs, sheep, and goats has been the focus of many past research studies, which applied traditional statistical regression techniques such as linear, multiple, and ridge regression, their success was limited by the multicollinearity and complex relationships among measurements (variables). To capture and explain such complex inter-variable relationships, a limited number of recent studies have reported the successful application of various ML and DL methods for predicting BW using features extracted from 2D and 3D digital images (Table 3). Successful applications of ML approaches on morphometric measurements extracted from 2D images via CV techniques have been reported in the literature for cattle.” Page 10, Right Column, Second Paragraph “Rudenko et al. (2020) applied ANNs and CV to identify the cow’s breed and estimate their BW. Cow images taken at different angles by synchronized cameras were fed to a Mask RCNN to determine the breed and position of each subject. Then, withers height, hip height, body length, and width of a cow were determined […] using the position of the cow detected by Mask RCNN in the previous step. Finally, the obtained data about the species and its size were fed to a multilayer perceptron (MLP) to estimate the live weight of the animals” – The images go through the Mask-RCNN (or Keypoint-RCNN of Patil from the combination in claim 1) including sending feature maps through the Keypoint branch/Mask branch which includes convolutional layers (as illustrated in Patel) and is sent to the multilayer-perceptron (which is fully connected) of Wang to determine a pig weight.)
Claim 7
Regarding claim 7, Wang in view of Patil, He, and Liu teaches the features of claim 1 and further teaches:
wherein in the step S4, the body size data comprises: a shoulder width, a hip width, and a body length, and the body size data is calculated according to a distance between the keypoints. (Wang Pages 5-7, Table 1, includes Shoulder Width (Page 7), Hip Width (Page 6), Body Length (Page 5); Page 11, Left Column, First Paragraph “The polar shape descriptors were measured as the distance from the centroid of the pig to points on its boundary contour. – The specific measurements are taught, and it is taught that measurements can be based on a distance between keypoints.)
Claim 8
Regarding claim 8, Wang in view of Patil, He, and Liu teaches the features of claim 1 and further teaches:
PNG
media_image11.png
332
136
media_image11.png
Greyscale
wherein in the step S3, the correcting is to correct the slanted pig with a minimum circumscribed rectangle. (Wang Page 8, Left Column, Third Paragraph “Postprocessing steps could include black and white or color-based masking, rotation, scaling and coordinate transformations” – Images are rotated to be aligned with the other pig images. Boxes are drawn around elements in both Mask-RCNN and Keypoint-RCNN and are optimized to be minimal, which is what is represented in the Box Offsets outputs of Patil.)
Claim 9: Wang in view of Patil, He, Liu, and Qiao2
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over NPL: “ASAS-NANP SYMPOSIUM: Applications of machine learning for livestock body weight prediction from digital images” by Wang et al. (Wang) in view of NPL “Human Pose Estimation using Keypoint RCNN in PyTorch” by Patil et al. (Patil), NPL: “Mask R-CNN” by He et al. (He), NPL: “Mask R-CNN” by He et al. (He), and NPL: “Video analytic system for detecting cow structure” by Liu et al. (Liu), and NPL: “Individual Cattle Identification Using a Deep Learning Based Framework” by Qiao et al. (Qiao2).
Claim 9
Regarding claim 9, Wang in view of Patil, He, and Liu teaches the features of claim 1. Wang in view of Patil, He, and Liu appear to fail to explicitly teach but Wang in view of Patil, He, Liu, and Qiao2 further teaches:
wherein in the step S4, the body size data and the weight data of the pig are bound with an identity of the pig and stored after being estimated, and the identity of the pig is obtained by recognizing back features of the pig in the images. (Qiao2 Page 318, Introduction “Cattle identification, which refers to the process of accurately recognizing individual cattle, plays an important role in automatic behavior analysis, weighing, health monitoring and welfare evaluation in precision livestock farming (Banhazi et al., 2012; Berckmans, 2014; McCabe et al., 2019). Once an animal is identified the growth, body weight can be tracked across time to achieve desirable outputs (He et al., 2016; Halachmi and Guarino, 2016). – In order to track the growth (size) and body weight of the pig, the pig’s weight and body size must be associated in data with identifying information. Page 319, Right Column, Second Paragraph “Recently, deep learning approaches with powerful feature extraction and image representation abilities have been widely used in the fields of visual classification and recognition (Tompson et al., 2014; Shen et al., 2019). After detecting trunks from raw images, Zhao and He (2015) proposed a CNN network method for cow identification. Kumar et al. (2018) proposed a CNN based approach for identification of individual cattle by using primary muzzle point image pattern. Zin et al. (2018) trained a CNN based on back images of cows to identify individual cows.” – The identification information is based on back features of the pig.)
Claim 10: Wang in view of Patil, He, Liu, and Truong
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over NPL: “ASAS-NANP SYMPOSIUM: Applications of machine learning for livestock body weight prediction from digital images” by Wang et al. (Wang) in view of NPL “Human Pose Estimation using Keypoint RCNN in PyTorch” by Patil et al. (Patil), NPL: “Mask R-CNN” by He et al. (He), NPL: “Mask R-CNN” by He et al. (He), and NPL: “Video analytic system for detecting cow structure” by Liu et al. (Liu), and NPL: “Loss functions: Why, what, where or when?” by Truong (Truong).
Claim 10
Regarding claim 10, Wang in view of Patil, He, and Liu teaches the features of claim 4 and further teaches
wherein the weight estimation model comprises a loss function, . (Patil Page 15, Loss Function in Keypoint-RCNN “As in Keypoint Detection, each Ground-Truth keypoint is one-hot-encoded, across all the K channels, in the featuremap of size [K=17, 56, 56], for a single object. For each visible Ground-Truth, channel wise Softmax (instead of sigmoid), from the final featuremap [17, 56, 56], is used to minimize the Cross Entropy Loss.)
Wang in view of Patil, He, and Liu teaches using the common cross entropy loss model, but appears to fail to explicitly teach, but Wang in view of Patil, He, Liu, and Truong teaches:
wherein the weight estimation model comprises a loss function, and the loss function uses a root mean square error function. (NOTE: RMSE loss and cross-entropy loss are two of the most commonly used loss functions, with cross-entropy loss being used more for classification problems (e.g., as in the example in Patil) and RMSE loss used more for regression problems of continuous quantity estimates (e.g., for training a model to infer pig weight based on image data). (Page 5 “Regression predictions can be evaluated using root mean squared error, whereas classification predictions cannot.” - The output of the pig weight model is the weight of a pig, which is a prediction of a continuous quantity, a regression)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claims to modify the cross-entropy loss function of the neural network in Patil as an element of the pig weight estimation model of Wang by the root mean square error of Truong because the person of ordinary skill in the art would be motivated to use a regression style model loss, such as root mean square error, for training a pig weight model with output of a continuous variable (i.e., pig weight), rather than the classification style loss function used in the Patil reference appropriately for a classification output (e.g., locations and characteristics of keypoints in segmented regions). This is because the person of ordinary skill in the art is motivated to ensure the training of the pig weight model of Wang is conducted meaningfully and efficiently using the selection of the root mean square error loss function in Truong for training a model to estimate the continuous quantity of pig weight rather that the cross-entropy loss function of Patil that Patil used for a classification problem. (Wang Page 3, Right Column, Middle Paragraph “The fourth approach based on CV and DL (CV+DL approach) represents a first step toward the full automation of the BW prediction process using digital images. The DL modeling component typically includes image selection, morphometric feature extraction, and feature selection as part of complex neural network architectures such as convolutional neural networks—CNNs (Fukushima, 1980), recurrent convolutional neural networks—RCNNs/RNNs (Spoerer et al., 2017), recurrent attention models—RAMs (Mnih et al., 2014), and RAMs with CNNs (Ba et al., 2014). Preliminary livestock studies implementing this approach reported significant improvements for BW prediction when compared with more traditional approaches (Fernandes et al., 2019, 2020a; Gjergji et al., 2020), nevertheless there is plenty of space for improvements” Patil “Loss Function in Keypoint-RCNN As in Keypoint Detection, each Ground-Truth keypoint is one-hot-encoded, across all the K channels, in the featuremap of size [K=17, 56, 56], fora single object. For each visible Ground-Truth, channel wise Softmax (instead of sigmoid), from the final featuremap [17, 56, 56], is used to minimize the Cross Entropy Loss.” Page 15 labels “A tensor of size [N], depicting the class of the object.” (Classification problem) Truong Page 4, So, what…? “there is not a single Loss Function that works for all kind of data. It depends on a number of factors including the presence of outliers, algorithm, time efficiency of gradient descent, ease of finding the derivatives and confidence of predictions… or maybe an accident. They can be categorized into 2 groups: Classification and Regression Loss. And of course, they are different. Classification is used for predicting a discrete class label and Regression is the task of predicting a continuous quantity.” Page 5 “Importantly, the way that we evaluate classification and regression predictions varies and does not overlap, for example: - Classification predictions can be evaluated using accuracy, whereas regression predictions cannot. - Regression predictions can be evaluated using root mean squared error, whereas classification predictions cannot.” Page 7 “Cross-Entropy Loss (or Log Loss) It measures the performance of a classification model whose output is a probability value between 0 and 1. Cross-entropy loss increases as the predicted probability diverges from the actual label.”) (Additionally, the substitution of the cross-entropy loss function for the root mean square error loss function would result in a combination of prior art elements according to known methods to yield predictable results, would be obvious to try among a finite list of common (often with premade functions in programming languages) loss functions, and would be an obvious substitute, all based on the demonstrated evidence and explanation.”)
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
(From A Prior Action)
US 20200394413 A1 to Bhanu et al. (Teaches using Mask R-CNN to generate keypoints)
US 20210037785 A1 to Yabe (Teaches estimating pig body weight based on image analysis using ML)
US 10962404 B2 to Kamiyama et al. (Teaches estimating weight based on photos using ML)
CN 108764210 A to Fang et al.(Teaches pig weight estimation based on image data using ML)
CN 113627486 A to Yang et al. (Teaches pig weight estimation based on image data)
NPL: “How to Choose Loss Functions When Training Deep Learning Neural Networks” by Brownlee (Teaches choosing MSE (related to RMSE) for training ML model gradient/continuous output)
NPL: “Imaging technologies to study the composition of live pigs: a review” by Carabus et al. (Teaches computer vision methods for determining pig body composition)
NPL: “Deep Learning Techniques for Beef Cattle Body Weight Prediction” by Gjergji et al. (Teaches using image data in ML to predict livestock bodyweight from keypoints)
NPL: “On-barn pig weight estimation based on body measurements by a Kinect v1 depth camera” by Pezzuolo et al. (Teaches using machine learning to determine pig body weight based on keypoints detected with a Kinect)
NPL “Multi-Pig Part Detection and Association with a Fully-Convolutional Network” by Psota et al. (Teaches determining specific parts of pigs using Mask R-CNN)
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JAY MICHAEL WHITE whose telephone number is (571)272-7073. The examiner can normally be reached Mon-Fri 11:00-7:00 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Ryan Pitaro can be reached at (571) 272-4071. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/J.M.W./Examiner, Art Unit 2188 /RYAN F PITARO/Supervisory Patent Examiner, Art Unit 2188