Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Examiner Note
Sometimes the specification refers to “evenly-sized” tiles and other times it refers to “evenly-spaced” tiles. The examiner believes that both are referring to the same tiles (otherwise, there could be new matter issue because the amendments recite “spaced,” but the identified support recites “sized.”)
Response to Arguments
Applicant’s arguments and amendments are persuasive. New issues are raised below.
Claim Objections
Claim 10 is objected to because of the following informalities:
The word “for” was deleted from the last line of claim 10.
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 1-4, 6-10, 12, 14, 15, and 17-19 (all claims) are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
Claims 1, 14, and 17 recite “a reference feature vector representing at least one reference image for the onboard model,” but having only one reference image is new matter. See, e.g., specification [0032] and [0033] “set of reference embeddings for that model,” [0035] “a model's reference embeddings are a sample of image embeddings that represent the full extent of imagery that the model was trained on,” and [0037] “a model's reference embeddings are a sample of image embeddings.”
Dependent claims are likewise rejected.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4, 6-10, 12, 14, 15, and 17-19 (all claims) are rejected under 35 U.S.C. 103 as being unpatentable over U.S. Pat. Pub. 20230196201 (“Imig”) and Lin PW, Hsu CM. Lightweight Convolutional Neural Networks with Model-Switching Architecture for Multi-Scenario Road Semantic Segmentation. Applied Sciences. 2021 Aug 12;11(16):7424 (“Lin”).
An apparatus, comprising:
a first device in a constrained-environment device, the first device including at least one memory having processor-executable code stored therein, and at least one processor that is adapted to execute the processor-executable code, wherein the processor-executable code includes processor-executable instructions that, in response to execution, enable the first device to perform actions, including, on the constrained- environment device: (Imig, Fig. 1)
storing image metadata and raw image data obtained from sensors on the constrained-environment device, wherein the raw image data includes a plurality of images; (Imig, [0050] “In one example, the input could be imagery (e.g., satellite imagery).”)
storing two or more onboard models; (Imig, [0029] “one or more models are disposed in an edge device”)
providing a plurality of image tiles such that the plurality of image tiles includes, for each image of the plurality of images, evenly-spaced portions of the image; (Imig, [0033] “a single frame from video data.” The BRI of the claim does not require that the claimed tiles or portions of the image are smaller than the entire frame. Additionally, Lin Fig. 4 teaches the claimed evenly-spaced portions.)
via an embedding-generation model, generating a plurality of image embeddings based on the plurality of image tiles; (Imig, Fig. 15, “Receive information corresponding to a plurality of sensors associated with a plurality of edge devices” 1515)
using the plurality of image embeddings and the set of reference embeddings for each onboard model of the two or more onboard models, for each image tile in the plurality of image tiles, for each onboard model in the plurality of onboard models, determining whether the onboard model should be executed on the image tile; and (Imig, Fig. 15, “Select one or more models …” 1520)
for each image tile for which a determination is made that an onboard model of the plurality of onboard models should be executed on the image tile, executing that onboard model on that image. (Imig, Fig. 15, “Deploy the one or more model pipeline” 1535)
Imig is not relied on for the below claim language.
However, Lin teaches storing, for each onboard model of the two or more onboard models, at least one reference embedding, wherein the at least one reference embedding for each onboard model is a reference feature vector representing at least one reference image for the onboard model, such that the apparatus stores two or more reference feature vectors associated respectively with the two or more onboard models; (Lin, Fig. 7, Convolutional Neural Network Weather Classifier)
wherein each image embedding is a feature vector representing graphical content of a corresponding image tile of the plurality of tiles; (Lin, Fig. 7, “Road image segmentation result.” Fig. 7 shows using a CNN to segment images as Sunny, Cloudy, or Rainy and thus the images are matched to these.)
for each image tile of the plurality of image tiles, comparing the feature vector representing the graphical content of the image tile to each of the two or more reference feature vectors associated respectively with the two or more onboard models in a vector comparison operation to determine, for each of the two or more onboard models, whether the onboard model should be executed on the image tile; and (Lin, Fig. 7, Convolutional Neural Network Weather Classifier)
It would have been obvious to one of ordinary skill in the art, before the effective filing date, to combine Imig and Lin such that Imig’s AI Inference platform with Lin’s model switching so that Imig’s model switching capability can also switch based on climate of the road image (e.g., Lin, section 2, “in order to be able to cope with the effect of different time and climate on the road image, this paper proposes a model-switching architecture to solve various road conditions.”)
Based on the above, these are examples of “combining prior art elements according to known methods to yield predictable results.” MPEP 2143.
2. The apparatus of claim 1, wherein the constrained-environment device is at least one of: an Internet of Things device that is in a constrained environment, a satellite in orbit, a spacecraft, or a stationary platform. (Imig, [0029] “For example, the edge device is a computing device disposed on or integrated with a satellite, … .” Imig [0029] discloses a series of examples.)
3. The apparatus of claim 1, wherein the feature vectors representing the plurality of images tiles are feature vectors of floating point numbers. (Lin, section 4.2, “It used VGG16 as the … weather classifier (green)” VGG (visual geometry group) teaches the claimed floating point numbers. Additionally, floating point is a known substitute for non-floating point numbers in computations. MPEP 2144.06(II).)
4. The apparatus of claim 1, wherein the feature vectors representing the plurality of image tiles each have at least 256 dimensions. (Lin, section 4.4.1, “Each of them comprised 250 training images with a resolution of 1920 × 1080” and Fig. 4. Fig. 4 shows a 5x5 feature convolved to a 1x1 feature, and 1920x1080 divided by 5x5 is more than 256.)
6. The apparatus of claim 1, wherein the sensors on the constrained-environment device include at least one of a camera, a synthetic aperture radar, a thermal imaging sensor, a hyperspectral sensor, or a video sensor. (Imig, [0050] “the inputs could include videos … .” Imig [0050] includes more examples.)
7. The apparatus of claim 1, wherein the embedding-generation model includes at least one of an unsupervised representation learning model, a self-supervised representation learning technique, or a supervised representation learning technique. (Lin, Fig. 12, “Ground Truth.” Lin’s use of “ground truth” teaches he claimed supervision.)
8. The apparatus of claim 1, wherein the two or more onboard models include at least one of a plane detection model, a ship detection model, a building detection model, a cloud detection model, a methane detection model, an oil tank detection model, a car detection model, or a fire detection model. (Imig, [0072] “collect a video frame and detect one truck in the middle.” Imig’s truck detection teaches the claimed car detection. See also, e.g., [0109] “For example, an edge device is assigned with a task of detecting wildfire and is configured to process data collected from various sensors on the edge device and transmit an insight of whether wildfire is detected” or [0051]’s “ship detection.”)
9. The apparatus of claim 1, the actions further including, after executing that onboard model on that image, performing a downlink sooner than a scheduled downlink based on results of the execution of that onboard model on that image. (Imig, [0071]-[0072] Imig teaches that a flying AIP might detect a truck in a video frame and send that out (teaching the claimed performing a downlink). Imig also teaches that the AIP may send video, and that the video may not be sent when flying (i.e., downloading the video after landing teaches the claimed scheduled downlink).)
10. The apparatus of claim 1, wherein using the plurality of image embeddings and the two or more onboard models, for each image tile in the plurality of image tiles, for each onboard model in the two or more onboard models, determining whether the onboard model should be executed on the image tile includes comparing, in a vector space, the image embedding that corresponds to that image tile with the at least one reference embedding that onboard model. (Imig, Fig. 15, “Select one or more models …” 1520. Imig [0084] teaches the claimed vector space, see the input vector and output vector at the end of the paragraph.)
12. The apparatus of claim 1, wherein the constrained-environment device is an orbiting satellite. (Imig, [0029] “For example, the edge device is a computing device disposed on or integrated with a satellite, and the sensor is an orbiting sensor.”)
Claims 14 and 15 are rejected under the same rationale as the corresponding apparatus claims. Note that the mapping of claim 12 addresses the claimed orbiting satellite.
Claims 17-19 are rejected under the same rationale as the corresponding apparatus claims.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
US20230156826A1 – titled “Edge computing in satellite connectivity environments”
US20210342669A1 – titled “Method, system, and medium for processing satellite orbital information using a generative adversarial network”
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DAVID ORANGE whose telephone number is (571)270-1799. The examiner can normally be reached Mon-Fri, 9-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Gregory Morse can be reached at 571-272-3838. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DAVID ORANGE/Primary Examiner, Art Unit 2663
Prosecution Insights