DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of the Claims
Claims 1-20 are pending for examination.
Claims 1, 8 and 15 are independent Claims.
Claims 1-20 are rejected under 35 U.S.C. §103.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ramos et al. (U.S. 2024/0167962 hereinafter Ramos) in view of Radhakrishman et al. (U.S. 2022/0327318 hereinafter Radha) in further view of McMahan et al. (U.S. 2019/0227980 hereinafter McMahan).
As Claim 1, Ramos teaches a method of monitoring infrastructure, comprising:
capturing video of infrastructure (Ramos (¶0094), edge devices collect monitory data such as videos) with at least one camera mounted on at least one moving vehicle (Ramos (¶0095 line 1-6), vehicle traverses the road network, the cameras of the edge device capture images of the pavement surface); and
processing the captured video and storing the video in a memory of at least one edge among a plurality of edge devices (Ramos (¶0117), “Report generating system 220 is configured to receive either raw or processed data from the edge devices 208 and/or pavement distress analysis system 218 (e.g., in real-time or near real-time) and may, in tum, generate one or more reports of pavement condition”), the at least one edge device device comprising a processor (Ramos (¶0207 line 2-4 fig. 9A item 908a, ¶0208, fig. 9A item 910a), sensor data segment is identified and extracted);
generating an inference of damage to the infrastructure and a severity thereof based on images in the captured video (Ramos (¶0077 line 1-5, ¶0116 line 1-16), pavement distress analysis system receives visual data and generate pavement distress data using a trained model. Pavement condition output includes PCI. PCI include distress type and severity level) and in response to running the inference locally on at least one edge device (Ramos (¶0120 line 1-5), pavement distress analysis system 218 and report generation system 220 are housed inside one or more of edge devices).
displaying the inference and associated metadata via a user interface (Ramos (¶0118 line 1-5), system generates reports indicating all pavement distress data in association with various pavement segments of a road network. Ramos (¶0171 last 3 lines), the output includes data (e.g. metadata) the type and severity of pavement distresses) associated with the at least one edge device, wherein the inference and the associated metadata are displayed via the user interface (Ramos (¶0120 line 1-5), pavement distress analysis system 218 and report generation system 220 are housed inside one or more of edge devices); and
Ramos may not explicitly disclose:
wherein the inference is generated using a compressed object detection and classification model created by structured pruning of a baseline model trained on publicly available data, the structured pruning removing low-weight filters based on a weight magnitude- based criterion model, wherein the structured pruning removes filters from convolutional layers of the baseline model to reduce computational resource utilization of the at least one edge device while maintaining local inference generation at the at least one edge device;
Radha teaches:
wherein the inference is generated using a compressed object detection and classification model created by structured pruning of a baseline model (Radha (¶0082 line 1-3 and 16-23), neural network for action recognition undergoes the pruning processes. Pruning processes calculates L1-norm values of kernels and removes kernels with L1-norm values below a defined threshold. Radha (¶0089 line 1-11), kernels includes plurality of weights. L1-norm and L2 norm are calculated based on kernel weights) trained on publicly available data (Radha (¶0593), model is trained on public data), the structured pruning removing low-weight filters based on a weight magnitude- based criterion model (Radha (¶0082 line 1-3 and 16-23), neural network for action recognition undergoes the pruning processes. Pruning processes calculates L1-norm values of kernels and removes kernels with L1-norm values below a defined threshold), wherein the structured pruning removes (Radha (¶0082 line 16-23), “pruning processes may include any suitable pruning process, such as those involving pruning weights, kernels, and/or variations thereof, and based on any suitable logic, heuristic, rules, functions, and/or variations thereof. In some examples, pruning is performed by a system by calculating L1-norm values of kernels of one or more neural networks of a system for action recognition 102, and removing kernels with L1-norm values below a defined threshold”) filters (Radha (¶0087 last 6 lines), “a filter may comprise a set of kernels. A single kernel that forms a filter may also be referred to as a filter.”) from convolutional layers of the baseline model (Radha (¶0117 line 1-5, line 9-10, deep convolution network (DCN)), “The system may utilize various action recognition models (e.g., action recognition models 126 of FIG. 1) to process cropped movements and cropped frames”) to reduce computational resource utilization of the at least one edge device while maintaining local inference generation at the at least one edge device (Radha (¶0082 last 8 lines), “A pruned neural network may utilize less computing resources to process data than computing resources that an unpruned neural network utilizes to process data, as the pruned neural network may comprise less kernels than kernels that the unpruned neural network comprises, and therefore may require less computing resources to process data than computing resources that the unpruned neural network requires to process data.”);
Ramos suggests that edge device captures and analyzes, using a machine learning model, infrastructure images for defects. Radha discloses a system/method for pruning machine learning models. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify of neural network of Ramos instead be a pruning processing taught by Radha, with a reasonable expectation of success. The motivation would be to allow “a pruned neural network may utilize less computing resources to process data than computing resources that an unpruned neural network” (Radha (¶00082 last 8 lines)).
Ramos in view of Radha may not explicitly disclose:
transmitting the inference and metadata to a remote server for participation in a federated learning process that aggregates model updates from a plurality of edge devices without transferring the captured video, wherein the federated learning process utilizes distributed data generated at the plurality of edge devices while raw captured video remains locally stored at the respective edge devices.
McMahan teaches:
transmitting the inference and metadata (McMahan (¶0032 line 4-8), updates includes sample C and expected weight for sample C) to a remote server for participation in a federated learning process (McMahan (¶0018 line 4-end), only update can be transmitted to the server) that aggregates model updates from a plurality of edge devices without transferring the captured video (McMahan (¶0054 line 4-8 and last 5 lines), training data is stored on client device. Only updates are combined to apply to the shared model. McMahan (¶0128 line 5-7), “The training data 108 can include audio files, image files, video files, a typing history, location history, and/or various other suitable data”), wherein the federated learning process utilizes distributed data generated at the plurality of edge devices while raw captured video remains locally stored at the respective edge devices (McMahan (¶0054 line 4-8 and last 5 lines), “in federated learning, a shared model can be trained while leaving the training data on each user's client computing device (e.g., mobile device). For example, devices can download the current model and compute an update by performing local computation on a local dataset”. McMahan (¶0018 line 4-7), “user data that is privacy sensitive can remain at the user's computing device and need not be uploaded to the server. Rather, only the less sensitive model update can be transmitted.”).
Ramos suggests that edge device captures and analyzes infrastructure images for defects. McMahan discloses a system/method to use images to train a local machine model. The system of McMahan further protects the image privacy by not transmitting these images and transmits only model updates to the central server. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify of neural network of Ramos in view of Radha instead be a federated learning taught by McMahan, with a reasonable expectation of success. The motivation would be provide “a user level differential privacy guarantee” (McMahan (¶0054 last 5 lines)).
As Claim 2, besides Claim 1, Ramos in view of Radha in further view of McMahan teaches wherein the running of the inference locally on the at least one edge device (Ramos (¶0120 line 1-5), pavement distress analysis system 218 and report generation system 220 are housed inside one or more of edge devices) comprises using a compression of models to run the inference on the at least one edge device (Ramos (¶0077 line 1-5, ¶0116 line 1-16), pavement distress analysis system receives visual data and generate pavement distress data using a trained model. Pavement condition output includes PCI) with a low computational resource (Ramos (¶0110 line 4-7), edge device is characterized as having low processing and computational power);and
the weight magnitude-based criterion comprise L2 norm (Radha (¶0089 line 1-11), kernels includes plurality of weights. L1-norm and L2 norm are calculated based on kernel weights).
As Claim 3, besides Claim 1, Ramos in view of Radha in further view of McMahan teaches:
enabling privacy preserved learning when generating the inference and the severity thereof by using distributed data subject to at least one federated learning framework of a split-federated learning framework of the federated learning process (McMahan (¶0030 last 4 lines), local update can be encoded before being provided to the one or more server computing devices).
As Claim 4, besides Claim 1, Ramos in view of Radha in further view of McMahan teaches wherein the at least one edge device includes the at least one camera mounted on at least one vehicle of a public transportation fleet of vehicles (Ramos (¶0112 line 7-10), edge devices can be mounted to existing fleet of vehicles), wherein the at least one vehicle comprises the at least one moving vehicle (Ramos (¶0095 line 1-6), vehicle traverses the road network, the cameras of the edge device capture images of the pavement surface).
As Claim 5, besides Claim 4, Ramos in view of Radha in further view of McMahan teaches further comprising capturing the location of the damage based on a position of the at least one vehicle (Ramos (¶0119 line 1-5), location data can be used to localize the pavement distress identified in the road network. Ramos (¶0112 line 7-10), edge devices can be mounted to existing fleet of vehicles).
As Claim 6, besides Claim 1, Ramos in view of Radha in further view of McMahan teaches further comprising displaying data indicative of the inference of the damage to the infrastructure in a cartographic display (Ramos (¶0147 line 1-4, fig. 5A), fig. 5A shows road network map that includes PCI values determined for each pavement segment in the road network).
As Claim 7, besides Claim 1, Ramos in view of Radha in further view of McMahan teaches further comprising: distributing training data among a plurality of clients, wherein the training data utilized in generating the inference of damage (Ramos (¶0164 last 6 lines, fig. 7B item 700b), training data are distributed to multiple edge device processors).
As Claims 8, Ramos teaches a system for monitoring infrastructure, comprising: at least one image-capturing device for capturing video of infrastructure (Ramos (¶0154 last 3 lines, ¶0180 line 5-7), image data are captured by a camera); and at least one edge device that communicates with the at least one image-capturing device (Ramos (¶0155 line 1-6), image data is analyzed by an edge device processor),
The rest of the limitation of are rejected for the same reasons as Claim 1.
As Claims 9-14, the Claims are rejected for the same reasons as Claims 2-7, respectively.
As Claims 15, Ramos teaches a system of monitoring infrastructure, comprising: at least one processor and a memory, the memory storing instructions to cause the at least one processor (Ramos (¶0122 line 1-4), edge device includes a processor and memory) to perform:
The rest of the limitations are rejected for the same reasons as Claim 1.
As Claim 16-20, the Claims are rejected for the same reasons as Claims 2-6, respectively.
Response to Arguments
I. Claim Rejections – 35 U.S.C. §101:
Applicants’ arguments are persuasive; therefore 35 U.S.C. §101 rejections on the Claims are respectfully withdrawn.
II. Claim Rejections – 35 U.S.C. §103:
As Claim 1, Applicants argue that cited references fail to teach or suggest the amended claims (first paragraph of page 12 in the remarks).
PNG
media_image1.png
256
678
media_image1.png
Greyscale
Applicants’ arguments are moot because new reference cited references teaches the amended limitation(s). Please see the current rejection(s) for details.
As Claim 1, Applicants argue that Office Action’s rejections improperly reply on hindsight reconstruction using selectively extracted concepts from unrelated references (first paragraph of page 12 in the remarks and paragraphs 4-6 of page 14 in the remarks).
PNG
media_image2.png
92
644
media_image2.png
Greyscale
Applicants’ arguments are not fully considered but are not persuasive. These rejections are based on motivations provided by the prior art(s). These prior art references are combined with each other by the motivation statement(s). Because these motivations are provided by the prior art(s), it is permissible hindsight.
As Claim 1, Applicants argues that Radha does not teaches “structured pruning removes filters ..” (third paragraph of page 13 in the remarks).
PNG
media_image3.png
116
674
media_image3.png
Greyscale
Applicants’ arguments are not fully considered but are not persuasive. Radha teaches the limitation(s). Please see the current rejection(s) for details.
As Claim 1, Applicants argue that McMahan does not teach or suggest retaining raw infrastructure-monitoring video locally while transmitting only metadata/model updates (third paragraph of page 14 in the remarks).
PNG
media_image4.png
146
680
media_image4.png
Greyscale
Applicants’ arguments are not fully considered but are not persuasive. Ramos suggests that edge device captures and analyzes infrastructure images for defects. McMahan discloses a system/method to use images to train a local machine model. The system of McMahan further protects the image privacy by not transmitting these images and transmits only model updates to the central server. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify of neural network of Ramos in view of Radha instead be a federated learning taught by McMahan, with a reasonable expectation of success. The motivation would be provide “a user level differential privacy guarantee” (McMahan (¶0054 last 5 lines)).
As Claim 1, Applicant argues that there is no reason to combine the references (first paragraph of page 15 in the remarks).
PNG
media_image5.png
204
652
media_image5.png
Greyscale
Applicant’s arguments are fully considered but are not persuasive. The statements to combine are copied below.
Ramos combine with Radha is “Ramos suggests that edge device captures and analyzes, using a machine learning model, infrastructure images for defects. Radha discloses a system/method for pruning machine learning models. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify of neural network of Ramos instead be a pruning processing taught by Radha, with a reasonable expectation of success. The motivation would be to allow “a pruned neural network may utilize less computing resources to process data than computing resources that an unpruned neural network” (Radha (¶00082 last 8 lines)).”
Ramos in view of Radha combine with McMahan is “Ramos suggests that edge device captures and analyzes infrastructure images for defects. McMahan discloses a system/method to use images to train a local machine model. The system of McMahan further protects the image privacy by not transmitting these images and transmits only model updates to the central server. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify of neural network of Ramos in view of Radha instead be a federated learning taught by McMahan, with a reasonable expectation of success. The motivation would be provide “a user level differential privacy guarantee” (McMahan (¶0054 last 5 lines)).”
Other dependent/independent Claims are not allowable for the same reasons above.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Chang et al. (U.S. 2022/0121949) discloses a system/method to prune a general model for personalized neural ne.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NHAT HUY T NGUYEN whose telephone number is (571)270-7333. The examiner can normally be reached M-F: 12:00-8:00 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Viker Lamardo can be reached at 571-270-5871. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/NHAT HUY T NGUYEN/Primary Examiner, Art Unit 2147