DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Specification
The disclosure is objected to because of the following informalities:
Paragraph 3 -> Extraneous “[“
Paragraph 50 -> “328” should be “318” ; See Fig. 4, there is no 328.
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 1 line 11: “a client device” should be “the client device” or “a second client device”.
Claim 1 recites “accessing an image captured by a client device operated by a user;
generating a set of structure lines based on the image by applying a computer-vision model to the accessed image, wherein the computer-vision model is a machine-learning model trained to generate structure lines for an image;
accessing a structure model for an area around a location of the client device at a time when the image was captured;
predicting a pose of the client device when the image was captured by comparing the set of structure lines to the structure model using an objective function, wherein the objective function is a function that generates an output that represents a likelihood that a client device is at a particular pose based on a set of structure lines from an image captured by the client device and a structure model;”
Should the accessing a structure model step should be placed before the accessing an image step? In order to access an image, it inherently must be captured first. It is physically impossible to access an image that has not been captured, excluding time travel. Therefore this claim limitation appear to be out of order, which is ok.
Under the same reasoning, The predicting a pose step should also be performed before the accessing image step. The problem here is that the predicting step requires the output of the machine-learning model (the set of structure lines for the image), which has not been generated yet. It is not clear how one can compare the structure lines before they have been generated. Note: predicting is done “when the image was captured” per the claim.
Claim 1 lines 6 and line 12 both recite “a structure model”. It is not clear if they are the same model or different models.
Claim 3 recites “when the client device was captured”. It is not clear what Applicant intends. Does applicant intent “when the client device captures the image”?
Claim 3 recites “wherein the objection function generates an output based on the sensor data”. It is not clear if the “sensor data” is in addition to the set of “structure lines” (in claim 1) or replacing the structure lines or some other variation.
Claim 4 recites “substantially linear structure”. Paragraph 70 defines substantially as +/- 10 percent. It is unclear what +/-10 percent of a straight line would mean.
Claim 7 recites “the semantic segment model”. This limitation lacks antecedent basis. Should be segmentation model?
Claim 11 is rejected under similar grounds as claim 1.
Claim 13 is rejected under similar grounds as claim 3.
Claim 14 is rejected under similar grounds as claim 4.
Claim 17 is rejected under similar grounds as claim 7.
Claims 2-10 and 12-20 are rejected as dependent upon a rejected claim.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Armagan (“Learning to Align Semantic Segmentation and 2.5D Maps for Geolocalization”) in view of Reimayr (“Going out: Robust Model-based Tracking for Outdoor Augmented Reality”).
Armagan discloses 1. A method comprising:
accessing an image captured by a client device operated by a user; (Armagan, Section 3,
PNG
media_image1.png
252
702
media_image1.png
Greyscale
)
generating a set of structure lines based on the image by applying a computer-vision model to the accessed image, wherein the computer-vision model is a machine-learning model trained to generate structure lines for an image; (Aramagan, Section 3.1,
PNG
media_image2.png
268
710
media_image2.png
Greyscale
)
accessing a structure model for an area around a location of the client device at a time when the image was captured; (Armagan, Section 3,
PNG
media_image1.png
252
702
media_image1.png
Greyscale
, 2.5D map )
predicting a pose of the client device when the image was captured by comparing the set of structure lines to the structure model using an objective function, wherein the objective function is a function that generates an output that represents a likelihood that a client device is at a particular pose based on a set of structure lines from an image captured by the client device and a structure model;(Armagan, Section 3.3,
PNG
media_image3.png
612
708
media_image3.png
Greyscale
, eqn 3 is the objective function, which is computed over the façade, horizontal edge, vertical edge and background, additionally see section 3.2 and Algorithm 1. The binary masks are created using the current pose * 2.5 Map; which is then compared ot the probability maps to determine the direction/orientation)
Armagan discloses the use in Augmented Reality (Section 1), but doesn’t explicitly disclose
“generating virtual content based on the predicted pose of the client device; and
displaying the virtual content on the client device.”
Reitmayr discloses
“generating virtual content based on the predicted pose of the client device; and
displaying the virtual content on the client device.”(reitmayr, fig. 1; additionally see abstract,
PNG
media_image4.png
292
592
media_image4.png
Greyscale
PNG
media_image5.png
396
1168
media_image5.png
Greyscale
)
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to postprocess the images of Aramagan with the AR process of Reitmayr.
The suggestion/motivation for doing so would have been Armagan expressly discloses AR as an intended use of their algorithm.
Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results.
Therefore, it would have been obvious to combine Armagan with Reitmayr to obtain the invention as specified in claim 1.
Armagan in view of Reitmayr discloses 2. The method of claim 1, further comprising: accessing a plurality of images captured by the client device, wherein the plurality of images comprises the accessed image; and generating the set of structure lines based on the plurality of images. (Armagan, Fig. 5)
Armagan in view of Reitmayr discloses 3. The method of claim 1, further comprising: accessing sensor data describing a pose of the client device at a time when the client device was captured; (Armagan, Abstract, GPS) and predicting the pose of the client device based on the objective function, wherein the objective function generate an output based on the sensor data. (see claim 1)
Armagan in view of Reitmayr discloses 4. The method of claim 1, wherein a structure line of the set of structure lines represents a substantially linear structure depicted by the image. (Armagan, Fig 1,Building)
Armagan in view of Reitmayr discloses 5. The method of claim 1, wherein a structure line of the set of structure lines represents a boundary between structures depicted by the image. (Armagan, Fig. 1b)
Armagan in view of Reitmayr discloses 6. The method of claim 1, wherein the computer-vision model is trained to identify sets of structures within images. (see claim 1)
Armagan in view of Reitmayr discloses 7. The method of claim 1, wherein the computer-vision model comprises a semantic segmentation model and wherein generating the set of structure lines comprises generating the set of structure lines based on segments generated by the semantic segment model. (see claim 1)
Armagan in view of Reitmayr discloses 8. The method of claim 1, wherein generating the set of structure lines comprises: identifying a type for each of a set of structures depicted in the image. (see claim 1, Façade, Horizontal Edge, Vertical Edge and Background)
Armagan in view of Reitmayr discloses 9. The method of claim 8, wherein the set of structure lines comprise an indication of a type of structure associated with a structure of the set of structures. (see claim 8)
Armagan in view of Reitmayr discloses 10. The method of claim 1, wherein the virtual content comprises augmented-reality content. (see claim 1)
Claims 11-20 are rejected under similar grounds as claims 1-10 above.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to GANDHI THIRUGNANAM whose telephone number is (571)270-3261. The examiner can normally be reached M-F 8:30-5PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sumati Lefkowitz can be reached at 571-272-3638. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/GANDHI THIRUGNANAM/ Primary Examiner, Art Unit 2672
Prosecution Insights