Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 7-9, 11, and 17-20 are rejected under 35 U.S.C. 103 as being unpatentable over Meier (EP 2770783) in view of Lee ("Object detection with sliding window in images including multiple similar objects," 2017 International Conference on Information and Communication Technology Convergence (ICTC), Jeju, Korea (South), 2017, pp. 803-806) and further in view of Sulk (US 20160155011).
Regarding claim 11, Meier teaches a computer apparatus comprising:
A processor (Paragraph 94, host processor); and
A memory storing instructions that, when executed by the processor (Paragraph 16, hardwired logic or configurable logic which processes logical functions), configure the apparatus to:
Access a camera image (Paragraph 53, image capture) generated by an optical sensor (Paragraph 55, capture device) of an augmented reality (AR) device (Paragraph 2, augmented reality);
Access a query image (Paragraph 30, reference image) from a storage of the AR device (Paragraph 30, second capturing device, Paragraph 2, augmented reality);
Maintain the camera image at original, unscaled resolution (Paragraph 61, In order to extract features at a certain scale, either the sampling window can be scaled accordingly or the image is scaled before computing the response of the feature detector); Meier teaches the case where only the size of the sampling window is scaled and the original image is not scaled.
Scale a detector window of a feature detector program to a first scaled size of a first scaled detector window (Paragraph 61, the sampling window can be scaled);
Extract features from the unscaled camera image by scanning the unscaled camera image with the first scaled detector window (Paragraph 61, In order to extract features at a certain scale, either the sampling window can be scaled accordingly or the image is scaled before computing the response of the feature detector); Meier teaches the case where only the size of the sampling window is scaled and the original image is not scaled.
Compare descriptors based on the extracted features from the camera image with descriptors based on the extracted features from the query image (Paragraph 61, In order to extract features at a certain scale, either the sampling window can be scaled accordingly or the image is scaled before computing the response of the feature detector; Paragraph 10, a feature descriptor is determined in order to enable the comparison and matching of features); and
Identify, using the feature detector program, the query image in the camera image based on the comparison of the descriptors (Paragraph 61, In order to extract features at a certain scale, either the sampling window can be scaled accordingly or the image is scaled before computing the response of the feature detector; Paragraph 10, a feature descriptor is determined in order to enable the comparison and matching of features).
While Meier fails to disclose the following, Lee teaches:
Maintaining the query image at original, unscaled resolution (Page 3, Paragraph 1, Rh denotes the height of a row in the bookshelf, and np denotes the number of book pages. The thickness of a book is estimated as 1 mm per every 10 pages.); Lee teaches maintaining the scale of the query image by matching the size of the sliding window to the query image.
Extract features from the unscaled query image by scanning the unscaled query image with a second unscaled detector window, wherein the second unscaled detector window is of a different size than the first scaled detector window (Page 3, Paragraph 1, For example, if the heights of an image and a bookshelf with 5 rows are 1000 pixels and 2000 mm, the height of a window is computed as 100 pixels; Page 3, Paragraph 3, In a given image, the feature points are detected in each sliding window). Note: The size of the second unscaled detector window is different than the size of the first scaled detector window because Meier teaches choosing the size of the scaled detector window.
Meier and Lee are both considered to be analogous to the claimed invention because they are in the same field of augmented reality. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Meier by using Lee to maintain the scale of the query image and extract features from the unscaled query image using an unscaled detector. Doing so would allow for detecting predetermined desired features in the query image.
While the combination of Meier and Lee fails to disclose the following, Sulc teaches:
Extract features without generating an image pyramid of the camera image (Paragraph 4, A sliding window may be used to scan a large set of possible candidate windows. In this approach, a window is moved stepwise across the image in fixed increments so that a decision is computed for multiple overlapping windows. In practice, this approach uses windows of different sizes and aspect ratios to detect objects at multiple scales, with different shapes, and from different viewpoints. Consequently, millions of windows are tested per image. The computational cost is, therefore, one of the major impediments to practical detection systems; Paragraph 30-32, produce good product candidate regions (or “windows”)… The localization technique(s) may rely on keypoint detection, keypoint description, and keypoint matching to obtain an accurate localization of objects 12 in the image. For example, a set of candidate regions 44 that are predicted to be locations of objects 12 of interest, is identified… A feature extraction component 46 extracts a region descriptor 48 from each of the candidate regions).
Sulc and the combination of Meier and Lee are both considered to be analogous to the claimed invention because they are in the same field of augmented reality. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the combination of Meier and Lee by using Sulc to use scaled detector windows to extract features without using an image pyramid. Doing so would allow for saving computation cost when determining the detector windows.
Method claim 1 and CRM claim 20 correspond to apparatus claim 11. Therefore, claims 1 and 20 are rejected for the same reasons as used above.
Regarding claim 17, the combination of Meier, Lee, and Sulc teaches the computing apparatus of claim 11, wherein the camera image includes a low resolution image (Meier, Paragraph 4, the camera image may take a low resolution image).
Method claim 7 corresponds to apparatus claim 17. Therefore, claim 7 is rejected for the same reasons as used above.
Regarding claim 18, the combination of Meier, Lee, and Sulc teaches the computing apparatus of claim 11. While the combination as presented previously fails to disclose the following, Lee further teaches:
Wherein the feature detector program includes ORB (Oriented FAST and Rotated BRIEF) local feature detector (Page 3, Paragraph 3, an ORB detector is adopted).
Lee and the combination of Meier and Sulc are both considered to be analogous to the claimed invention because they are in the same field of augmented reality. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the combination of Meier and Sulc by using Lee to use ORB. Doing so would consider trade-off between the accuracy and the speed in feature detection (Lee, Page 3, Paragraph 3).
Method claim 8 corresponds to apparatus claim 18. Therefore, claim 8 is rejected for the same reasons as used above.
Regarding claim 19, the combination of Meier, Lee, and Sulc teaches the computing apparatus of claim 11, wherein the feature detector program is configured to compare descriptors based on extracted features from the camera image using a scaled detector with descriptors based on extracted features from the query image using an unscaled detector (Meier, Paragraph 61, In order to extract features at a certain scale, either the sampling window can be scaled accordingly or the image is scaled before computing the response of the feature detector; Paragraph 10, a feature descriptor is determined in order to enable the comparison and matching of features).
Method claim 9, corresponds to apparatus claim 19. Therefore, claim 9 is rejected for the same reasons as used above.
Claims 5 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Meier view of Lee and further in view of Sulc as applied to claims 1, 7-9, 11, and 17-20 above and further in view of Bean (US 9940692).
Regarding claim 15, the combination of Meier, Lee, and Sulc teaches the computing apparatus of claim 11. While it fails to disclose the following, Bean teaches:
Wherein the instructions further configure the apparatus to:
Access a virtual content item corresponding to the query image (col 1, Summary, identifying a first element of interest within the first image... associating a corresponding first AR content overlay for the first element of interest); and
Display the virtual content item in a display of the AR device (col 3, paragraph 4, Information (e.g., content overlays) associated with an element of interest can be displayed (e.g., in a callout, in a window, etc.) on an AR display).
Bean and the combination of Meier, Lee, and Sulc are both considered to be analogous to the claimed invention because they are in the same field of augmented reality. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the combination of Meier, Lee, and Sulc by using Bean to determine an AR content overlay depending on a predetermined image. Doing so would provide users AR overlays correlated to specific query images, customizing their AR experience.
Method claim 5 corresponds to apparatus claim 15. Therefore, claim 5 is rejected for the same reasons as used above.
Claims 6 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Meier view of Lee and further in view of Sulc as applied to claims 1, 7-9, 11, and 17-20 above and further in view of Reed (US 20180284454).
Regarding claim 16, the combination of Meier, Lee, and Sulc teaches the computing apparatus of claim 11. While it fails to disclose the following, Reed teaches:
Wherein the instructions further configure the apparatus to: generate the camera image using a wide-angle lens coupled to the optical sensor (Paragraph 82, the optical component may be a wide angle lens).
Reed and the combination of Meier, Lee, and Sulc are both considered to be analogous to the claimed invention because they are in the same field of augmented reality. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the combination of Meier, Lee, and Sulc by using Reed to incorporate a wide-angle camera. Doing so would allow the system to enlarge the field of view of the camera (Reed, paragraph 82).
Method claim 6 corresponds to apparatus claim 16. Therefore, claim 6 is rejected for the same reasons as used above.
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Meier view of Lee and further in view of Sulc as applied to claims 1, 7-9, 11, and 17-20 above and further in view of Davidson (WO 2022254409).
Regarding claim 10, the combination of Meier, Lee, and Sulc teaches the method of claim 1. While it fails to disclose the following, Davidson teaches:
Wherein the query image includes marker data that indicates a pre-defined visual code (Paragraph 181, The reference features may be a quick response (QR) code or known exemplar or marker, which can provide processor 310 certain information).
Davidson and the combination of Meier, Lee, and Sulc are both considered to be analogous to the claimed invention because they are in the same field of augmented reality. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the combination of Meier, Lee, and Sulc by using Davidson to a pre-defined visual code (QR code) as the query image. Doing so would allow the query image to more easily identifiable in the camera image.
Response to Amendment
Applicant’s arguments with respect to claims 1, 11, and 20 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument. Sulc teaches the limitation “without generating an image pyramid of the camera image” by addressing the high computation cost of generating an image pyramid and by using keypoint matching and determining candidate regions of the camera image from which to extract features.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SNIGDHA SINHA whose telephone number is (571)272-6618. The examiner can normally be reached Mon-Fri. 12pm-8pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jason Chan can be reached at 571-272-3022. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SNIGDHA SINHA/Examiner, Art Unit 2619
/JASON CHAN/Supervisory Patent Examiner, Art Unit 2619