DETAILED ACTION
This Non-Final Office Action is responsive to the claims filed on March 27, 2026.
Claims 1 and 4 are objected to.
Claims 1, 4, and 6-10 is rejected under 35 USC 112(a).
Claims 1, 4, and 6-10 are rejected under 35 USC 112(b).
Claims 1, 4, and 6-10 are allowable over art.
Response To Arguments/Amendments
35 USC 112(a): The Applicant’s arguments and amendments have been considered and are persuasive. Accordingly, the rejection is withdrawn.
35 USC 112(b): The Applicant’s arguments and amendments have been considered and are persuasive for most of the issues. The following issue remains.
The Keypoint-RCNN Algorithm Is Added With A Keypoint Branch On The Basis Of A Mask-RCNN: This appears to be an awkward translation, and, contrary to the Applicant’s naked assertion that the amendments to claim 1 clarified this, its metes and bounds were not further clarified by the amendment. Please revisit the rejection for the reasons asserted by the examiner that the Applicant can choose to attempt to rebut.
35 USC 101: The Applicant’s arguments and amendments have been considered and are persuasive. Accordingly, the rejection is withdrawn.
35 USC 103: The Applicant’s arguments and amendments have been considered and are persuasive. Accordingly, the rejection is withdrawn.
Claim Objections
Claims objected to because of the following informalities:
The claims are replete with confusing language and redundancy, and are completely disorganized. The Examiner will attempt to point out as many formal issues with the claims as possible, but this may need to be dealt with in more than one round of action/response.
With regard to claim structure/organization, claim 1 recites steps S1-S4 and then recites a number of qualifiers that qualify the steps S1-S4 mixed in with further steps. This explicitly includes qualifying elements that are part of the step S4 prior to and after the Step S2. For purposes of clarity and to improve the value of the claims the Applicant should place all of the elements currently presented in the claims after the steps S1-S4, distributed among/appropriately registered among the steps S1-S4. Indentation can also help this clarification. For example,
1. A method for estimating a body size and weight of a pig based on deep learning, the method comprising:
S1, obtaining the images of the pig from video frames;
S2, detecting, using a keypoint detection algorithm, keypoints of the pig in the images, to obtain a keypoint detection result, and removing images of incomplete pigs from the images according to the keypoint detection result to generate images of complete pigs, wherein:
in the step S2, by using an instance segmentation algorithm, the images are subjected to instance segmentation first before the keypoints are detected, and pixels belonging to the pig in the images are marked; the instance segmentation algorithm is built based on a Mask RCNN instance segmentation network; and an instance segmentation process comprises:
inputting the images into the ResNext-101 feature extraction network in the Mask RCNN instance segmentation network to obtain a feature map;
setting a fixed number of regions of interest for each pixel position of the feature map, inputting the regions of interest into a region proposal network in the Mask RCNN instance segmentation network to perform binary classification to obtain a foreground and a background, and performing coordinate regression, so as to obtain final regions of interest;
performing ROIAlign operation on the obtained regions of interest, namely, first establishing a correspondence between pixels of the original images and the feature map and then establishing a correspondence between the feature map and fixed features; and
classifying the regions of interest in a fully connected layer, generating detection boxes of detected objects in the regions of interest, and performing regression on the regions of interest to make the detection boxes gradually approach correct positions of the detected objects, and performing segmentation in a fully convolutional layer, to finally obtain a result of instance segmentation,
wherein the instance segmentation algorithm is run in a server, and after each instance segmentation, the segmented images of the pig being retained in the server;
the keypoint detection algorithm in the step S2 is built based on a Keypoint-Recurrent Convolutional Neural Network (Keypoint-RCNN) algorithm and is configured to detect the keypoints, wherein
the keypoint detection algorithm is configured to detect and mark the keypoints, the position of each keypoint is modeled as a separate one-hot mask, each type of keypoint has a mask, and only one pixel for each keypoint is marked as the foreground; and
the keypoints obtained by segmentation comprise: a left ear root point, a right ear root point, a left front elbow point, a right front elbow point, a left rear elbow point, a right rear elbow point, a spinal back point, and a tail root point;
the Keypoint-RCNN algorithm is added with a keypoint branch on the basis of a Mask-RCNN, and a feature extraction network of the Keypoint-RCNN algorithm adopts the ResNext-101 feature extraction network;
S3, detecting that the pig is slanted in a portion of the images of complete pigs, and aligning the portion of the images of complete pigs in which the pig is determined to be slanted to generate images of aligned and complete pigs;
S4, inputting the images of aligned and complete pigs into a weight estimation model to determine the weight of the pigs in the images of aligned and complete pigs,
wherein the weight estimation model in the step S4 is built based on a ResNext-101 feature extraction network and is trained by:
preparing a training data set comprising a plurality of the images of the pig and the weight of the pig corresponding to each image, segmenting the pig of each image in the training data set, and binarizing the images to obtain binarized images of the pig and the weight of the pig corresponding to the images; and
dividing the training dataset into training set, a test set and a validation set according to a ratio of 6:2:2, inputting the training set into the weight estimation model to perform model training to determine model parameters, then testing, by the test set, an estimation accuracy of the weight estimation mode, and finally inputting the validation set into the weight estimation model to further adjust the model parameters, as to obtain a trained weight estimation model; and
S5, calculating body size data according to the keypoint detection result.
Claim 1 recites, “video frams.” This appears to be a typo.
Claim 1 recites, “a current frame image is revoked.” Revoked is an awkward term and does not sufficiently convey what happens to the current frame image. For purposes of examination, this will be interpreted to mean that the current frame image is removed from a certain group if images (though it is unclear from which group of claimed images the current frame image is removed).
Claim 1 recites, “if the pig’s angle is standard, […].” Standard is a relative term for which no standard has been provided.
Claim 1 recites, “the instance segmentation algorithm ran in a server.” This appears to be a typo.
Claim 1 recites, “the weight estimation model is subject to model training after being built; and a training process is as follows:“ This appears intended to link the claimed model training with the training process but fails to do so explicitly.
Claim 1 recites, “dividing the training dataset into training set.” “Training set” lacks antecedent basis.
Claim 1 recites, “S4, inputting the images into a weight estimation model” and “dividing […] as to obtain a trained weight estimation model.” It is unclear what the relationship is between the weight estimation model in S4 and the trained weight estimation model of the last dividing limitation.
Claim 4 recites, “wherein in the step S4, the weight estimation model is built based on the ResNext-101 feature extraction network,” but this is also redundantly recited in claim 1.
Appropriate correction is required. No new matter may be added.
Claim Rejections - 35 USC § 112(a)
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 1, 4, and 6-10 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the enablement requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to enable one skilled in the art to which it pertains, or with which it is most nearly connected, to make and/or use the invention.
The Keypoint-RCNN Algorithm Is Added With A Keypoint Branch On The Basis Of A Mask-RCNN
As indicated in the following 35 USC 112(b) rejection, this limitation is being interpreted to mean the network shown in the Applicant’s FIG. 4. This is problematic, however, because the Box Offsets and Class Scores for Keypoint-RCNN and Mask-RCNN differ in dimension and nature. This means that the fully connected layers that output the box offsets and class scores would have to be different, and the specification fails to enable and describe this and fails to illustrate this. (See the Patil et al. reference of record that demonstrates the differences in dimensionality; Also See the newly added Zheng reference that was published a month and a half after the first priority document in this case which shows the contemporaneous difficulty of the art, specifically, discussing the difficulty of modifying an existing Keypoint-RCNN branch of the Keypoint-RCNN model on Page 151, Left Column, Third Paragraph – Page 152, Right Column, First Paragraph, and Compare Fig. 1 on page 151 to Fig. 2 on page 152.). Further, FIGs. 3 and 4 include elements stated as “RPN” and “FPN,” both of which are essential structural elements of the networks in FIGs. 3 and 4. However, these abbreviations are not described anywhere in the specification. A person of ordinary skill in the art would not deem the definitions of these abbreviations readily apparent without reference to some source that defined the abbreviations. That is, the network illustrated in Applicant’s FIG. 4 is not enabled, described, or correctly illustrated by this specification and likely incorrectly depicts the flow of information (e.g., that the outputs from the ROIAlign layer would be the same for all of the Keypoint branch, Mask branch, and Fully Connected layer) that would be necessary to attach keypoint and mask branches to the same foundation that includes a common backbone and ROIAlign layer. The following is an assessment of the limitation against the most significant Wands factors from MPEP 2164.01:
(B) The nature of the invention: Much of the claim includes standard machine learning features and specific data elements for determining the weight and size of a pig. The less conventional features of the invention include the use of a Keypoint-RCNN network, which is a variant of a Mask-RCNN. It is also unconventional to have both Mask-RCNN and Keypoint-RCNN branches in the same model.
(C) The state of the prior art: The prior art does not appear to teach a network with both a Mask branch and a Keypoint-RCNN branch extending from the same network elements, including undefined RPN and FPN elements. The adaptation a Keypoint-RCNN network to modify its branches was on the cutting edge, even a month and a half after the filing of the priority document in this case. This is demonstrated with respect to the Patil and Zheng references described above.
(D) The level of one of ordinary skill: The art is the art of determining pig properties using machine learning. This means the expertise of the person of ordinary skill in the art would be divided between expertise in pig size estimation and expertise in machine learning to determine models to be used and how to deploy the models. This person likely possesses a bachelors or masters in machine learning. This person likely lacks the skill to adapt a Mask-RCNN network to add a Keypoint-RCNN branch, especially in light of the teaches of the Patil and Zheng references described above.
(E) The level of predictability in the art: The predictability of using one or the other of a Mask-RCNN or Keypoint-RCNN is known. However, the connection of these as two branches in the same model requires custom model building and an understanding of the dimensional differences of the inputs for a Keypoint-RCNN and a Mask-RCNN. This makes this element of the art unpredictable.
(F) The amount of direction provided by the inventor: The inventor provided only a single network diagram for the embodiment of FIG. 4. Further, the inventor provided no description for FIG. 4 other than the brief description of the drawing, which states, “FIG. 4 is a network structure diagram of keypoint detection algorithm.” Further still, FIG. 4 includes the elements, RPN and FPN, which are not defined in the FIGs., claims, written description, or any other part of the specification.
(G) The existence of working examples: The only example provided is the network in FIG. 4, which, as discussed in element F, has no corresponding description and also lacks definitions for the RPN and FPN, which are integral elements of the models in the claim.
(H) The quantity of experimentation needed to make or use the invention based on the content of the disclosure: Without knowing what RPN and FPN are the person of ordinary skill in the art will never know, regardless of the amount of experimentation, whether the person has made or infringed on the invention. That is, the person of ordinary skill in the art could experiment infinitely and never be sure that the person has made and used the invention.
Dependent claims depending from rejected claims are rejected based on their dependency.
Claim Rejections - 35 USC § 112(b)
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1, 4, and 6-10 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
The Keypoint-RCNN Algorithm Is Added With A Keypoint Branch On The Basis Of A Mask-RCNN
Claim 1 recites, “the Keypoint-RCNN algorithm is added with a keypoint branch on the basis of a Mask-RCNN.” This language is unclear. The claim language sounds like a poor translation and does not mean anything. It appears that the claim language was intended to claim the model as illustrated in the Applicant’s FIG. 4. However, based on the current language, it is broad enough to state a generic Keypoint-RCNN (e.g., without an added Mask Branch), which includes a keypoint branch with keypoint head and mask, and its basis is a Mask-RCNN (because the Keypoint-RCNN is a variation of and derivation from the Mask-RCNN). However, given the new amendments and the limited disclosure of the specification, for purposes of examination, the feature will be examined as if it is reciting the network shown in FIG. 4 of the Applicant’s specification. If the Applicant wants to claim both the Mask branch and an attached Keypoint branch, as illustrated in the Applicant’s FIG. 4, there must be a clear recitation in the claim that both branches are attached within the same model, e.g., stemming from the same elements.
Regardless of the asserted interpretation used during examination, appropriate correction by amendment of the indefinite elements is required. Dependent claims depending from rejected claims are rejected based on their dependency.
Allowability Over Art
Claims 1, 4, and 6-10 are allowable over the prior art. The following is a statement of reasons for allowance over the art:
Independent claim 1 recites the following distinguishing feature:
[…] the weight estimation model in the step S4 is built based on a ResNext-101 feature extraction network, and the Keypoint-RCNN algorithm is added with a keypoint branch on the basis of a Mask-RCNN […]
The primary Wang reference of record (NPL: “ASAS-NANP SYMPOSIUM: Applications of machine learning for livestock body weight prediction from digital images” by Wang et al.) teaches (in bold) teaches the use of most of the machine learning techniques in the claim for livestock weight and size estimation. This includes Mask-RCNN, a variant of which is Keypoint-RCNN. With respect to independent claim 1, Wang teaches:
A method for estimating a body size and weight of a pig based on deep learning, the method comprising the following steps:
S1, obtaining images of the pig;
S2, detecting, by using a keypoint detection algorithm, keypoints of the pig in the images to obtain a keypoint detection result, and removing images that the pig is incomplete in the images and retaining images that the pig is complete in the images according to the keypoint detection result;
S3, detecting whether the pig is slanted in the images, and correcting the images of the slanted pig to obtain images that the pig is complete and not slanted in the images; and
S4, inputting the images into a weight estimation model and calculating body size data according to the keypoint detection result to obtain the weight and body size data of the pig,
wherein the keypoint detection algorithm in the step S2 is built based on [Mask-RCNN] algorithm, the weight estimation model in the step S4 is built based on [a Mask-RCNN algorithm], whether the pig in the images is complete is judged according to the keypoint detection result, and
if the pig in the image is not complete, a current frame image is revoked and a next frame image is obtained,
if the pig in the image was complete, it is judged whether a pig’s angle is standard and that the pig is not slanted, if the pig’s angle is standard, the images of the pig are judged to be qualified images of the pig, and if the pig was slanted, a slant angle of the pig in the images is corrected to determine the qualified images of the pig, and
the instance segmentation algorithm ran in a server, and after each instance segmentation, the images of the pig of multiple video key frames would be retained in the server;
wherein in the step S2, by using an instance segmentation algorithm, the images are subjected to instance segmentation first before the keypoints are detected, and pixels belonging to the pig in the images are marked; :
the weight estimation model is subject to model training after being built; and a training process is as follows:
preparing a training data set comprising a plurality of the images of the pig and the weight of the pig corresponding to each image, segmenting the pig of each image in the training data set, and binarizing the images to obtain binarized images of the pig and the weight of the pig corresponding to the images; and
dividing the training data set into a training set, a test set and a validation set according to a ratio of 6:2:2, inputting the training set into the weight estimation model to perform model training to determine model parameters, then testing, by the test set, an estimation accuracy of the weight estimation model, and finally inputting the validation set into the weight estimation model to further adjust the model parameters, so as to obtain a trained weight estimation model.
The Patil reference of record (NPL “Human Pose Estimation using Keypoint RCNN in PyTorch” by Patil et al.) teaches the relationship between Mask-RCNN and Keypoint-RCNN and shows an example of estimating pose of a human using the Keypoint-RCNN variant of the Mask-RCNN. With respect to independent claim 1, Patil teaches:
wherein the keypoint detection algorithm in the step S2 is built based on Keypoint-Recurrent Convolutional Neural Network (Keypoint-RCNN) algorithm, and is configured to detect the keypoints of received video frames,
the weight estimation model in the step S4 is built based on and the Keypoint-RCNN algorithm is added with a keypoint branch on the basis of a Mask-RCNN,.
the instance segmentation algorithm is built based on a Mask RCNN instance segmentation network; and an :
inputting the images into the ResNext-101 feature extraction network in the Mask RCNN instance segmentation network to obtain a feature map;
setting a fixed number of regions of interest for each pixel position of the feature map, inputting the regions of interest into a region proposal network in the Mask RCNN instance segmentation network to perform binary classification to obtain a foreground and a background, and performing coordinate regression, so as to obtain high-quality regions of interest;
performing ROIAlign operation on the obtained regions of interest, namely, first establishing a correspondence between pixels of the original images and the feature map and then establishing a correspondence between the feature map and fixed features; and
classifying the regions of interest in a fully connected layer, generating detection boxes of detected objects in the regions of interest, and performing regression on the regions of interest to make the detection boxes gradually approach correct positions of the detected objects, and performing segmentation in a fully convolutional layer, to finally obtain a result of instance segmentation.
wherein in the step S2, the keypoints are detected after the instance segmentation, and the keypoint detection algorithm is built based on the Keypoint-RCNN algorithm; the keypoint detection algorithm is configured to detect and mark the keypoints, position of each keypoint is modeled as a separate one-hot mask, each type of keypoint has a mask, and only one pixel for each keypoint is marked as the foreground; and
the keypoints obtained by segmentation comprise: a left ear root point, a right ear root point, a left front elbow point, a right front elbow point,
The He reference of record (NPL: “Mask R-CNN” by He et al.) teaches (in bold) teaches using ResNext-101 as a backbone for a Mask-RCNN network, and in combination with Patil, using the same backbone for Keypoint-RCNN. With respect to independent claim 1, He teaches:
a ResNext-101 feature extraction network, and a feature extraction network of the Keypoint-RCNN algorithm adopts the ResNext-101 feature extraction network; and the weight estimation model in the step S4 uses the ResNext-101 feature extraction network
The Liu reference of record (NPL: “Video analytic system for detecting cow structure” by Liu et al.) teaches (in bold) specific keypoints identified that are not explicitly stated in the other references. With respect to independent claim 1, Liu teaches:
the keypoints obtained by segmentation comprise: a left ear root point, a right ear root point, a left front elbow point, a right front elbow point, a left rear elbow point, a right rear elbow point, a spinal back point, and a tail root point.
Independent claim 1 recites the combination of these features that use and train a network that detects keypoints to be used in weight and/or size determinations of a pig from images. The specified distinguishing feature interpreted herein as representing the network shown in the Applicant’s FIG. 4 that includes Mask-RCNN and Keypoint-RCNN branches from a common backbone and undefined RPN and FPN elements are not taught in the prior art references. The combination of these specific elements cannot be taught by the combination of the cited references without the use of impermissible hindsight. A further search was conducted, and no combination of references that teach the features of the independent claims without the use of impermissible hindsight was found.
Therefore, claims 1, 4, and 6-10, as drafted, are rendered neither obvious nor anticipated by the prior art of the record and the available field of prior art.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
(From A Prior Action)
US 20200394413 A1 to Bhanu et al. (Teaches using Mask R-CNN to generate keypoints)
US 20210037785 A1 to Yabe (Teaches estimating pig body weight based on image analysis using ML)
US 10962404 B2 to Kamiyama et al. (Teaches estimating weight based on photos using ML)
CN 108764210 A to Fang et al.(Teaches pig weight estimation based on image data using ML)
CN 113627486 A to Yang et al. (Teaches pig weight estimation based on image data)
NPL: “How to Choose Loss Functions When Training Deep Learning Neural Networks” by Brownlee (Teaches choosing MSE (related to RMSE) for training ML model gradient/continuous output)
NPL: “Imaging technologies to study the composition of live pigs: a review” by Carabus et al. (Teaches computer vision methods for determining pig body composition)
NPL: “Deep Learning Techniques for Beef Cattle Body Weight Prediction” by Gjergji et al. (Teaches using image data in ML to predict livestock bodyweight from keypoints)
NPL: “On-barn pig weight estimation based on body measurements by a Kinect v1 depth camera” by Pezzuolo et al. (Teaches using machine learning to determine pig body weight based on keypoints detected with a Kinect)
NPL “Multi-Pig Part Detection and Association with a Fully-Convolutional Network” by Psota et al. (Teaches determining specific parts of pigs using Mask R-CNN)
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JAY MICHAEL WHITE whose telephone number is (571)272-7073. The examiner can normally be reached Mon-Fri 11:00-7:00 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Ryan Pitaro can be reached at (571) 272-4071. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/J.M.W./Examiner, Art Unit 2188
/RYAN F PITARO/Supervisory Patent Examiner, Art Unit 2188
Prosecution Insights