DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of the Claims
Claims 1-20 are pending for examination.
Claims 1, 8 and 15 are independent Claims.
Claims 1-20 are rejected under 35 U.S.C. §103.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ramos et al. (U.S. 2024/0167962 hereinafter Ramos) in view of Radhakrishman et al. (U.S. 2022/0327318 hereinafter Radha) in further view of McMahan et al. (U.S. 2019/0227980 hereinafter McMahan).
As Claim 1, Ramos teaches a method of monitoring infrastructure, comprising:
capturing video of infrastructure (Ramos (¶0094), edge devices collect monitory data such as videos) with at least one camera mounted on at least one moving vehicle (Ramos (¶0095 line 1-6), vehicle traverses the road network, the cameras of the edge device capture images of the pavement surface); and
processing the captured video and storing the video in a memory of at least one edge device comprising a processor (Ramos (¶0207 line 2-4 fig. 9A item 908a, ¶0208, fig. 9A item 910a), sensor data segment is identified and extracted);
generating an inference of damage to the infrastructure and a severity thereof based on images in the captured video (Ramos (¶0077 line 1-5, ¶0116 line 1-16), pavement distress analysis system receives visual data and generate pavement distress data using a trained model. Pavement condition output includes PCI. PCI include distress type and severity level) and in response to running the inference locally on at least one edge device (Ramos (¶0120 line 1-5), pavement distress analysis system 218 and report generation system 220 are housed inside one or more of edge devices).
displaying the inference and associated metadata via a user interface (Ramos (¶0118 line 1-5), system generates reports indicating all pavement distress data in association with various pavement segments of a road network. Ramos (¶0171 last 3 lines), the output includes data (e.g. metadata) the type and severity of pavement distresses) associated with the at least one edge device, wherein the inference and the associated metadata are displayed via the user interface (Ramos (¶0120 line 1-5), pavement distress analysis system 218 and report generation system 220 are housed inside one or more of edge devices); and
Ramos may not explicitly disclose:
wherein the inference is generated using a compressed object detection and classification model created by structured pruning of a baseline model trained on publicly available data, the structured pruning removing low-weight filters based on a weight magnitude- based criterion model;
Radha teaches:
wherein the inference is generated using a compressed object detection and classification model created by structured pruning of a baseline model (Radha (¶0082 line 1-3 and 16-23), neural network for action recognition undergoes the pruning processes. Pruning processes calculates L1-norm values of kernels and removes kernels with L1-norm values below a defined threshold. Radha (¶0089 line 1-11), kernels includes plurality of weights. L1-norm and L2 norm are calculated based on kernel weights) trained on publicly available data (Radha (¶0593), model is trained on public data), the structured pruning removing low-weight filters based on a weight magnitude- based criterion model (Radha (¶0082 line 1-3 and 16-23), neural network for action recognition undergoes the pruning processes. Pruning processes calculates L1-norm values of kernels and removes kernels with L1-norm values below a defined threshold);
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify of neural network of Ramos instead be a pruning processing taught by Radha, with a reasonable expectation of success. The motivation would be to allow “a pruned neural network may utilize less computing resources to process data than computing resources that an unpruned neural network” (Radha (¶00082 last 8 lines)).
Ramos in view of Radha may not explicitly disclose:
transmitting the inference and metadata to a remote server for participation in a federated learning process that aggregates model updates from a plurality of edge devices without transferring the captured video.
McMahan teaches:
transmitting the inference and metadata (McMahan (¶0032 line 4-8), updates includes sample C and expected weight for sample C) to a remote server for participation in a federated learning process (McMahan (¶0018 line 4-end), only update can be transmitted to the server) that aggregates model updates from a plurality of edge devices without transferring the captured video (McMahan (¶0054 line 4-8 and last 5 lines), training data is stored on client device. Only updates are combined to apply to the shared model).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify of neural network of Ramos in view of Radha instead be a federated learning taught by McMahan, with a reasonable expectation of success. The motivation would be provide “a user level differential privacy guarantee” (McMahan (¶0054 last 5 lines)).
As Claim 2, besides Claim 1, Ramos in view of Radha in further view of McMahan teaches wherein the running of the inference locally on the at least one edge device (Ramos (¶0120 line 1-5), pavement distress analysis system 218 and report generation system 220 are housed inside one or more of edge devices) comprises using a compression of models to run the inference on the at least one edge device (Ramos (¶0077 line 1-5, ¶0116 line 1-16), pavement distress analysis system receives visual data and generate pavement distress data using a trained model. Pavement condition output includes PCI) with a low computational resource (Ramos (¶0110 line 4-7), edge device is characterized as having low processing and computational power);and
the weight magnitude-based criterion comprise L2 norm (Radha (¶0089 line 1-11), kernels includes plurality of weights. L1-norm and L2 norm are calculated based on kernel weights).
As Claim 3, besides Claim 1, Ramos in view of Radha in further view of McMahan teaches:
enabling privacy preserved learning when generating the inference and the severity thereof by using distributed data subject to at least one federated learning framework of a split-federated learning frame work of the federated learning process (McMahan (¶0030 last 4 lines), local update can be encoded before being provided to the one or more server computing devices).
As Claim 4, besides Claim 1, Ramos in view of Radha in further view of McMahan teaches wherein the at least one edge device includes the at least one camera mounted on at least one vehicle of a public transportation fleet of vehicles (Ramos (¶0112 line 7-10), edge devices can be mounted to existing fleet of vehicles), wherein the at least one vehicle comprises the at least one moving vehicle (Ramos (¶0095 line 1-6), vehicle traverses the road network, the cameras of the edge device capture images of the pavement surface).
As Claim 5, besides Claim 4, Ramos in view of Radha in further view of McMahan teaches further comprising capturing the location of the damage based on a position of the at least one vehicle (Ramos (¶0119 line 1-5), location data can be used to localize the pavement distress identified in the road network. Ramos (¶0112 line 7-10), edge devices can be mounted to existing fleet of vehicles).
As Claim 6, besides Claim 1, Ramos in view of Radha in further view of McMahan teaches further comprising displaying data indicative of the inference of the damage to the infrastructure in a cartographic display (Ramos (¶0147 line 1-4, fig. 5A), fig. 5A shows road network map that includes PCI values determined for each pavement segment in the road network).
As Claim 7, besides Claim 1, Ramos in view of Radha in further view of McMahan teaches further comprising: distributing training data among a plurality of clients, wherein the training data utilized in generating the inference of damage (Ramos (¶0164 last 6 lines, fig. 7B item 700b), training data are distributed to multiple edge device processors).
As Claims 8, Ramos teaches a system for monitoring infrastructure, comprising: at least one image-capturing device for capturing video of infrastructure (Ramos (¶0154 last 3 lines, ¶0180 line 5-7), image data are captured by a camera); and at least one edge device that communicates with the at least one image-capturing device (Ramos (¶0155 line 1-6), image data is analyzed by an edge device processor),
The rest of the limitation of are rejected for the same reasons as Claim 1.
As Claims 9-14, the Claims are rejected for the same reasons as Claims 2-7, respectively.
As Claims 15, Ramos teaches a system of monitoring infrastructure, comprising: at least one processor and a memory, the memory storing instructions to cause the at least one processor (Ramos (¶0122 line 1-4), edge device includes a processor and memory) to perform:
The rest of the limitation of are rejected for the same reasons as Claim 1.
As Claim 16-20, the Claims are rejected for the same reasons as Claims 2-6, respectively.
Response to Arguments
I. Claim Rejections – 35 U.S.C. §101:
Applicants’ arguments are persuasive; therefore 35 U.S.C. §101 rejections on the Claims are respectfully withdrawn.
II. Claim Rejections – 35 U.S.C. §102:
As Claim 1, Applicants argue that Ramos does not disclose “capturing video of infrastructure … on a moving vehicle” (sixth paragraph of page 15 in the remarks).
PNG
media_image1.png
275
663
media_image1.png
Greyscale
Applicants’ arguments are not persuasive. Ramos (¶0095 line 1-6) teaches that vehicle traverses the road network, the cameras of the edge device capture images of the pavement surface.
As Claim 1, Applicants argue that Ramos does not disclose “pruning of a baseline model …” (first paragraph of page 16 in the remarks).
PNG
media_image2.png
224
648
media_image2.png
Greyscale
Applicants’ arguments are moot because new reference Radha teaches the limitation(s).
As Claim 1, Applicants argue that Ramos does not disclose “displaying the inference …” (second paragraph of page 16 in the remarks).
PNG
media_image3.png
194
659
media_image3.png
Greyscale
Applicants’ arguments are not persuasive. Ramos (¶0118 line 1-5), system generates reports indicating all pavement distress data in association with various pavement segments of a road network. Ramos (¶0171 last 3 lines), the output includes data (e.g. metadata) the type and severity of pavement distresses.
As Claim 1, Applicants argue that Ramos does not disclose “transmitting the inference …” (third paragraph of page 16 in the remarks).
PNG
media_image4.png
197
653
media_image4.png
Greyscale
Applicants’ arguments are moot because new reference McMahan teaches the limitation(s).
As Claim 1, Applicants argue that Ramos does not disclose “model compression using … L2 norm” (fourth paragraph of page 17 in the remarks).
PNG
media_image5.png
114
647
media_image5.png
Greyscale
Applicants’ arguments are moot because new reference Radha teaches the limitation(s).
As Claim 4-9, 11-14, 15, Applicant argues that these Claims are allowable based on the arguments of the base Claims (pages 18-20).
Applicants’ arguments are not persuasive for the same reason(s) above.
III. Claim Rejections – 35 U.S.C. §102:
As Claim 3, 10 and 17, Applicants argue that Ramos does not disclose “a split-federated learning frame work” (first paragraph of page 23 in the remarks).
PNG
media_image6.png
229
666
media_image6.png
Greyscale
Applicants’ arguments are moot because new reference McMahan teaches the limitation(s).
Other dependent/independent Claims are not allowable for the same reasons above.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NHAT HUY T NGUYEN whose telephone number is (571)270-7333. The examiner can normally be reached M-F: 12:00-8:00 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Viker Lamardo can be reached at 571-270-5871. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/NHAT HUY T NGUYEN/Primary Examiner, Art Unit 2147
Prosecution Insights