DETAILED ACTION
Notice of Pre-AIA or AIA Status
This is in response to application no.19/283,148 filed on 07/28/2025. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-5, 10, 12-14, 16-18, 20 and 21 is/are rejected under 35 U.S.C. 103 as being unpatentable over Aljapur et al. (US 20230301280 A1) in view of Molchanov et al. (US 20180365532 A1).
Regarding claim 1, Aljapur teaches the claim as follows:
A method comprising: obtaining fish images from a camera device (¶0030: The system 400 includes video cameras and/or stereoscopic sensors 402 installed in an aquafarm 401…Images, video, and/or other information from the sensors 402 can be fed to on-board edge computing devices 403 which can operate one or more machine learning models for obtaining biomass estimations…based on information obtained from the sensors 402, including for example: fish detection…); generating predicted values by providing one or more of the fish images to an end-to-end model trained to estimate weight of fish from the fish images (¶0031:the computing devices and machine leaming models can determine biomass of a fish or other animal based on images, videos, or other data obtained from cameras or other sensors. In some instances, for example, the machine learning models can be trained to recognize certain desired frames or images of fish in a video feed…the machine leaming models or other system components can estimate a size of the fish, for example, a fish weight…), wherein the end-to-end model … configured to adjust one or more parameters of the end-to-end model (¶0029: The machine learning models can be used for a variety of purposes, including feed recognition, feed control…¶0030: for example, the overall biomass or biomass distribution for the aquafarm 401 can be used to determine or control an amount of feed provided to fish in the aquafarm (e.g., in a single feeding session). This can involve adjusting or controlling one or more feed parameters…); comparing the predicted values to ground truth data representing weights of one or more fish (¶0031: by comparing an expected shape or profile for the fish in a straight pose with images collected by the cameras or other sensors. An image of a fish that matches the expected shape can be identified as a desired image and can be used for further processing (e.g., for fish size …)the machine leaming models or other system components can estimate a size of the fish, for example, a fish weight…).
Aljapur does not explicitly teach wherein the end-to-end model comprises one or more differentiable layers…and updating the one or more parameters of the end-to-end model based on the comparison of the predicted values.
However, Molchanov teaches wherein the end-to-end model comprises one or more differentiable layers…(¶0038: Use of the soft-argmax layer 114 to extract landmark locations from pixel-level predictions makes the entire sequential multi-tasking system 135 differentiable and trainable end-to-end through back-propagation); and updating the one or more parameters of the end-to-end model based on the comparison of the predicted values (¶0038, 0045: Differences are computed between the predicted class labels (attributes) output by the classifier neural network model 130 and the GT attribute labels for the training dataset. Differences may also be computed between GT landmark labels (when available) and the predicted landmark locations output by the neural network model 220. The differences represent errors that are propagated backwards through the classifier neural network model 130 and used to update parameters…).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have modified Aljapur’s biomass estimation system by incorporating the teaching of Molchanov as noted above, in order to improve the accuracy of the neural network model (Molchanov: ¶0038, 0045).
Regarding claim 2, Aljapur teaches the method of claim 1, comprising: providing the predicted values of the end-to-end model to one or more devices (¶0035: One or more machine learning models are used (step 506) that receive the data as input and provide as output a determination of a fish biomass, a fish biomass distribution, and/or a fish satiation level for the aquaculture cage); and performing an action upon receipt of the predicted values, in which the action configures the one or more devices (¶0030, 0035: Based on the determination from the one or more machine learning models, an amount of feed delivered to the aquaculture cage from the feed supply is controlled (step 508)).
Regarding claim 3, Aljapur teaches the method of claim 2, wherein the action that configures the one or more devices further comprises adjusting a feeding system (¶0030: the overall biomass or biomass distribution for the aquafarm 401 can be used to determine or control an amount of feed provided to fish in the aquafarm (e.g., in a single feeding session). This can involve adjusting or controlling one or more feed parameters, such as an amount of food provided during a feed session, a rate at which food is provided over a period of time, and/or a frequency at which the fish are fed (e.g., a number of feed sessions per day or week)).
Regarding claim 4, Aljapur teaches the method of claim 1, wherein the predicted values include one or more values indicating a weight of a fish represented by the fish images (¶0031:the computing devices and machine leaming models can determine biomass of a fish or other animal based on images, videos, or other data obtained from cameras or other sensors. In some instances, for example, the machine learning models can be trained to recognize certain desired frames or images of fish in a video feed…the machine leaming models or other system components can estimate a size of the fish, for example, a fish weight…).
Regarding claim 5, Aljapur teaches the method of claim 1, wherein the fish images include two images from a pair of stereo cameras of the camera device (¶0030: The system 400 includes video cameras and/or stereoscopic sensors 402 installed in an aquafarm 401 located in an offshore environment. Images, video, and/or other information from the sensors 402 can be fed to on-board edge computing devices 403…information obtained from the sensors 402, including for example: fish detection…).
Regarding claim 10, Aljapur teaches the method of claim 1, wherein the ground truth data includes one or more values that represent a weight of at least one fish from the one or more fish (¶0031: by comparing an expected shape or profile for the fish in a straight pose with images collected by the cameras or other sensors. An image of a fish that matches the expected shape can be identified as a desired image and can be used for further processing (e.g., for fish size …) the machine leaming models or other system components can estimate a size of the fish, for example, a fish weight…).
Regarding claim 12, Aljapur teaches the method of claim 1, comprising: obtaining the ground truth data from a system that measures the one or more fish (0031: by comparing an expected shape or profile for the fish in a straight pose with images collected by the cameras or other sensors. An image of a fish that matches the expected shape can be identified as a desired image and can be used for further processing (e.g., for fish size …the machine leaming models or other system components can estimate a size of the fish, for example, a fish weight…).
Regarding claim 13, Aljapur in view of Molchanov teaches the method of claim 1. Molchanov further teaches wherein the end-to-end model is a convolutional neural network that comprises the one or more differentiable layers (¶0037-0038: the neural network model 110 includes six convolutional layers with 7×7 kernels, followed by two convolutional layers with 1×1 kernels, then the soft-argmax layer 114 for landmark localization…Use of the soft-argmax layer 114 to extract landmark locations from pixel-level predictions makes the entire sequential multi-tasking system 135 differentiable and trainable end-to-end through back-propagation). The motivation statement set forth above with respect to claim 1 applies here.
Regarding claim 14, Aljapur in view of Molchanov teaches method of claim 1. Molchanov further teaches wherein the comparison of the predicted values and the ground truth data comprises determining a regression error between the predicted values and a value of the ground truth data ( ¶0038: Differences may also be computed between GT landmark labels (when available) and the predicted landmark locations output by the neural network model 220. The differences represent errors that are propagated backwards through the classifier neural network model 130 and used to update parameters (e.g., weights) used by the neural network model 110 to predict the landmark locations (i.e., landmark coordinates)). The motivation statement set forth above with respect to claim 1 applies here.
Regarding claim 16, Aljapur in view of Molchanov teaches the method of claim 1. Aljapur further teaches wherein the end-to-end model is configured to generate an (¶0031:the computing devices and machine leaming models can determine biomass of a fish or other animal based on images, videos, or other data obtained from cameras or other sensors. In some instances, for example, the machine learning models can be trained to recognize certain desired frames or images of fish in a video feed…the machine leaming models or other system components can estimate a size of the fish, for example, a fish weight…). Moreover, Molchanov discloses input images are labeled with ground truth attribute class labels than are labeled with landmark locations…differences between the ground truth attribute class labels and the predicted attributes generated by the classifier neural network model 130 are used to compute adjusted parameters for the classifier neural network model. Molchanov, ¶0032, 0038. Accordingly, teaches the end-to-end model is configured to generate an output label as recited in claim 16. The motivation statement set forth above with respect to claim 1 applies here.
Regarding claim 17, Aljapur in view of Molchanov teaches the method of claim 16. Aljapur further teaches wherein the end-to-end model is configured to compare (¶0031: by comparing an expected shape or profile for the fish in a straight pose with images collected by the cameras or other sensors. An image of a fish that matches the expected shape can be identified as a desired image and can be used for further processing (e.g., for fish size …the machine leaming models or other system components can estimate a size of the fish, for example, a fish weight…). Moreover, Molchanov discloses input images are labeled with ground truth attribute class labels than are labeled with landmark locations…differences between the ground truth attribute class labels and the predicted attributes generated by the classifier neural network model 130 are used to compute adjusted parameters for the classifier neural network model. Molchanov, ¶0032, 0038. Accordingly, teaches wherein the end-to-end model is configured to compare the output label representing …to a corresponding label of the ground truth data, as recited in claim 17. The motivation statement set forth above with respect to claim 1 applies here.
Regarding claim 18, Aljapur in view of Molchanov teaches the method of claim 17. Molchanov further teaches wherein the end-to-end model is configured to update the one or more parameters of the model when the output label does not match the label of the ground truth data (¶0038: Differences are computed between the predicted class labels (attributes) output by the classifier neural network model 130 and the GT attribute labels for the training dataset. Differences may also be computed between GT landmark labels (when available) and the predicted landmark locations output by the neural network model 220...The parameters are updated to reduce the differences and improve accuracy). The motivation statement set forth above with respect to claim 1 applies here.
Regarding claim 20, the claim is drawn to a non-transitory computer-readable medium claim and recites the limitation analogous to claim 1, and is rejected due to a similar reason set forth above with respect to claim 1.
Regarding claim 21, most of the limitations of the claim are analogous to the limitations of claim 1, and are rejected due to a similar reason set forth above with respect to claim 1. Aljapur further teaches a system comprising: one or more processors (¶0030: The system 400 includes …Images, video, and/or other information from the sensors 402 can be fed to on-board edge computing devices 403); and machine-readable media interoperably coupled with the one or more processors and storing one or more instructions that, when executed by the one or more processors, perform operations comprising (¶0037-0040: The processor 610 is capable of processing instructions stored in the memory 620 or on the storage device 630).
Claim(s) 8-9 is/are rejected under 35 U.S.C. 103 as being unpatentable over Aljapur et al. (US 20230301280 A1) in view of Molchanov et al. (US 20180365532 A1) as applied to claim 1, and further in view of Erhan et al. (US 9373057 B1).
Regarding claim 8, Aljapur in view of Molchanov do not teach wherein identifying the one or more fish in each image of the two images further comprises generating one or more bounding boxes for each image, wherein the one or more bounding boxes represent an enclosed region of the respective image with an associated likelihood indicating presence of a fish.
However, Erhan teaches wherein identifying the one or more fish in each image of the two images further comprises generating one or more bounding boxes for each image, wherein the one or more bounding boxes represent an enclosed region of the respective image with an associated likelihood indicating presence of a fish (col. 3, lines 16-23: a system that can train a neural network that is configured to receive an input image and generate data defining a predetermined number of candidate bounding boxes within the input image and, for each candidate bounding box, a confidence score that represents the likelihood that the bounding box contains an image of an object).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have modified Aljapur’s biomass estimation system by incorporating the teaching of Erhan as noted above, in order to accurately determine candidate bounding boxes that the input image are likely to contain an image of an object (Erhan : col. 7, lines 1-10).
Regarding claim 9, Aljapur in view of Molchanov do not teach wherein computing the re-projection error between the corresponding two-dimension coordinate and the rectified two-dimensional coordinate further comprises: generating one or more rectified bounding boxes for the rectified image, wherein the one or more rectified bounding boxes represent an enclosed region of the rectified image with an associated likelihood indicating presence of a fish ; computing a detection score for each of the one or more rectified bounding boxes, wherein the detection score is based on the associated likelihood indicating presence of the fish ; and providing the detection score to the end-to-end model.
However, Erhan teaches wherein computing the re-projection error between the corresponding two-dimension coordinate and the rectified two-dimensional coordinate further comprises: generating one or more rectified bounding boxes for the rectified image, wherein the one or more rectified bounding boxes represent an enclosed region of the rectified image with an associated likelihood indicating presence of a fish (col. 3, lines 16-23: a system that can train a neural network that is configured to receive an input image and generate data defining a predetermined number of candidate bounding boxes within the input image and, for each candidate bounding box, a confidence score that represents the likelihood that the bounding box contains an image of an object); computing a detection score for each of the one or more rectified bounding boxes, wherein the detection score is based on the associated likelihood indicating presence of the fish (col. 3, lines 16-23: a system that can train a neural network that is configured to receive an input image and generate data defining a predetermined number of candidate bounding boxes within the input image and, for each candidate bounding box, a confidence score that represents the likelihood that the bounding box contains an image of an object); and providing the detection score to the end-to-end model (col. 4,lines 34-42: The bounding box data includes data that defines the predetermined number of candidate bounding boxes within the training image and the confidence score generated by the object detection neural network 102 for each candidate bounding box. The neural network training system 100 updates the current values of the parameters of the object detection neural network 102 using the bounding box data …)
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have modified Aljapur’s biomass estimation system by incorporating the teaching of Erhan as noted above, in order to accurately determine candidate bounding boxes that the input image are likely to contain an image of an object (Erhan : col. 7, lines 1-10).
Claim(s) 11 is/are rejected under 35 U.S.C. 103 as being unpatentable over Aljapur et al. (US 20230301280 A1) in view of Molchanov et al. (US 20180365532 A1) as applied to claim 1, and further in view of Shang et al. (US 20230267385 A1).
Regarding claim 11, Aljapur in view of Molchanov do not teach wherein the camera device is equipped with locomotion devices for moving within a fish pen.
However, Shang teaches wherein the camera device is equipped with locomotion devices for moving within a fish pen (¶0051: the housing of camera 202 may include or be attached to underwater propulsion mechanisms such as propellers or water jets).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have modified Aljapur’s biomass estimation system by incorporating the teaching of Shang as noted above, in order to allow the camera to capture images of the fish from different locations within the enclosure (Shang: ¶0050-0051).
Claim(s) 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Aljapur et al. (US 20230301280 A1) in view of Molchanov et al. (US 20180365532 A1) as applied to claim 1, and further in view of Novosad et al. (US 20220180524 A1).
Regarding claim 15, Aljapur in view of Molchanov do not teach wherein the end-to-end model is configured to update the one or more parameters of the model when the regression error exceeds a threshold value.
However, Novosad teaches wherein the end-to-end model is configured to update the one or more parameters of the model when the regression error exceeds a threshold value (¶0118-0120: The errors can be used during a procedure called backpropagation to update the parameters of the deep learning network…The errors can be compared to predetermined criteria, such as proceeding to a sustained minimum for a specified number of training iterations. If the errors do not satisfy the predetermined criteria, then model parameters of the deep learning model 708 can be updated using backpropagation…).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have modified Aljapur’s biomass estimation system by incorporating the teaching of Novosadas noted above, in order to reduce errors during the model training (Novosadas:¶0119).
Claim(s) 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Aljapur et al. (US 20230301280 A1) in view of Molchanov et al. (US 20180365532 A1) as applied to claim 1, and further in view of Zhang et al. (US 20190132576 A1).
Regarding claim 19, Aljapur in view of Molchanov do not teach wherein generating the rectified image comprises determining the combination of the two images based on intrinsic properties of the camera device.
However, Zhang teaches wherein generating the rectified image comprises determining the combination of the two images based on intrinsic properties of the camera device (¶0025: An image corrector and projector 112 can obtain calibrated camera intrinsic parameters from the intrinsic calibrator 110…The images may also be rectified and aligned so that pixels between each image pair are aligned along the same horizontal line. Rectification may be used to transform images by projecting two-or-more images onto a common image plane or spherical surface).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have modified Aljapur’s biomass estimation system by incorporating the teaching of Zhang as noted above, in order to align the images taken by cameras that may be misaligned (Zhang :¶0025).
Allowable Subject Matter
Claims 6-7 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is the prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Kobayashi (US 20230045358 A1) describes an underwater organism imaging aid device or the like that captures an image of an underwater organism in order to estimate the size of the underwater organism.¶0001.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NATHNAEL AYNALEM whose telephone number is (571)270-1482. The examiner can normally be reached M-F 9AM-5:30 PM ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, SATH PERUNGAVOOR can be reached at 571-272-7455. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/NATHNAEL AYNALEM/Primary Examiner, Art Unit 2488
Prosecution Insights