DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statements (IDS) submitted on 10/10/2024 and 10/10/2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statements are being considered by the examiner.
Allowable Subject Matter
Claims 28 and 31 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 20-23, 25-27, 30 is/are rejected under 35 U.S.C. 102(a) as being taught by Lim et al. (US Patent Number 10122969 -B1, hereinafter “Lim”)
Regarding claim 20, Lim teaches: A computer-implemented method for re-projecting and combining sensor data for visualization, the method comprising: receiving further sensor data from a plurality of sensors, the sensor data including images and depth information; ([0026], " The video capture system 100 includes a plurality of RGB (red-green-blue) cameras 110a, 110b, 110c, and 110d (collectively referred to as “RGB cameras 110”). Although the example illustrated in FIG. 1 includes four RGB cameras 110,"; [0027], "In some implementations, the video capture system 100 includes one or more depth cameras (collectively referred to as “depth cameras 115”). In some examples, some or all of the depth cameras 115 are positioned behind the display screen 105 to capture light for depth estimation through the display screen 105, such as is illustrated for the two depth cameras 115a and 115b in the example of FIG. 1.")
determining a background for each image; ([0029], "The term “background” may be abbreviated as “BG” in portions of this disclosure. In some implementations, a background camera may be selected based on at least a position of the subject in relation to the RGB cameras 110; for example, by identifying a camera in which the subject is expected to occupy little or none of the camera's FOV. In the example shown in FIG. 1, the RGB camera 110a has been selected as a background camera, and a background image 140a has been obtained from the selected RGB camera 110a. The background image 140a may span only a portion of a total FOV of the RGB camera 110a. In this particular example, the background image 140a includes images of the table 125 and the participants 134, 136, and 138, but does not show the subject 132.")
subtracting the determined background from the sensor data to produce background-subtracted images; performing blob detection on the background-subtracted images; ([0027], "As described in more detail below, depth estimates obtained using the depth cameras 115 may be used to, among other things, determine when a subject has come into proximity to the video capture system 100, determine a distance between the video capture system 100 and a subject, determine a position of a subject relative to one or more of the RGB cameras 110, and/or identify discontinuities in a depth image and related depth image data used to perform image segmentation for a subject.")
determining cropped images for each image in the sensor data, the cropped images comprising detected blobs with the background subtracted; communicating the cropped images to a projection module along with cropping information; ([0074], "The RGB image 630d captured by the selected foreground camera is received by the foreground segmenter 340, which, for the foreground subject 132, segments the RGB image 630d to identify a foreground portion 660 of the RGB image 630d (for example, by discriminating pixels included in the foreground portion 660 from background pixels 665). The segmentation may be performed based on an identification of pixels in the RGB image 630d that correspond to depth estimates included within the foreground portion 622 of the depth image 620b. The foreground image generator 345 generates a foreground image 670 for the foreground subject 132 by resizing (for example, using a proportional scaling) the foreground portion 660 from a total height 662 to a reduced total height 672.")
determining the position of the cropped images with respect to a combined view using the cropping information; ([0027], "A depth estimate may also be referred to as an “estimated depth,” “distance estimate,” or “estimated distance.” As described in more detail below, depth estimates obtained using the depth cameras 115 may be used to, among other things, determine when a subject has come into proximity to the video capture system 100, determine a distance between the video capture system 100 and a subject, determine a position of a subject relative to one or more of the RGB cameras 110, and/or identify discontinuities in a depth image and related depth image data used to perform image segmentation for a subject.")
overlaying the cropped images onto a static background combined image; ([0074], "FIG. 6C illustrates an example of in which a background image 645 and a foreground image 670 are generated and used to generate a composite image 690 for the scene 600 illustrated in FIGS. 6A and 6B. ")
and outputting the combined image with the overlaid cropped images to a display device for visualization. ([0086], "Computer system 1100 may be coupled via bus 1102 to a display 1112, such as a liquid crystal display (LCD), for displaying information. One or more user input devices, such as the example user input device 1114 can be coupled to bus 1102, and can be configured for receiving various user inputs, such as user command selections and communicating these to processor 1104, or to a main memory 1106. The user input device 1114 can include physical structure, or virtual implementation, or both, providing user input modes or options, for controlling, for example, a cursor, visible to a user through display 1112 or through other techniques, and such modes or operations can include, for example virtual mouse, trackball, or cursor direction keys.")
Regarding claim 21, Lim teaches: The method of claim 20, wherein the static background combined image is generated from previously received static combined images or blueprint/CAD images. ([0074], "FIG. 6C illustrates an example of in which a background image 645 and a foreground image 670 are generated and used to generate a composite image 690 for the scene 600 illustrated in FIGS. 6A and 6B. ")
Regarding claim 22, Lim teaches: The method of claim 20, wherein the background is determined by re-projecting sensor data without depth information and morphing the image to a plane. ([0074], "FIG. 6C illustrates an example of in which a background image 645 and a foreground image 670 are generated and used to generate a composite image 690 for the scene 600 illustrated in FIGS. 6A and 6B. ")
Regarding claim 23, Lim teaches: The method of claim 20, wherein the background is determined by re-projecting sensor data without depth information and morphing the image to a plane. ([0074], "FIG. 6C illustrates an example of in which a background image 645 and a foreground image 670 are generated and used to generate a composite image 690 for the scene 600 illustrated in FIGS. 6A and 6B. ")
Regarding claim 25, Lim teaches: The method of claim 20, wherein the cropped images are communicated to the projection module with cropping information comprising the four corners of the cropped region in the local coordinate space of the sensor. ([0074], "The RGB image 630d captured by the selected foreground camera is received by the foreground segmenter 340, which, for the foreground subject 132, segments the RGB image 630d to identify a foreground portion 660 of the RGB image 630d (for example, by discriminating pixels included in the foreground portion 660 from background pixels 665). The segmentation may be performed based on an identification of pixels in the RGB image 630d that correspond to depth estimates included within the foreground portion 622 of the depth image 620b. The foreground image generator 345 generates a foreground image 670 for the foreground subject 132 by resizing (for example, using a proportional scaling) the foreground portion 660 from a total height 662 to a reduced total height 672.")
Regarding claim 26, Lim teaches: The method of claim 20, wherein the combined image is updated periodically with a predetermined frequency to reflect changes in the sensor data. ([0064], "In some implementations, the video capture system 300 includes an image/video encoder 365 that encodes the frame image 362 as part of a series of frame images in a video stream. In some implementations, the video capture system 300 includes a video conferencing module 370 that is configured to establish and/or participate in a video conferencing session via network(s) 390 with one or more remote systems, such as remote display system 380 at a geographic location 384. The network(s) 390 may include, for example, one or more wired or wireless data communication networks, and/or the Internet. The video conferencing module 370 may be implemented as an application program executed by the video capture system 300. In some implementations, the video capture system 300 may include a virtual webcam module (not illustrated in FIG. 3) configured to appear as a webcam or other video camera to application programs, including, for example, the video conferencing module 370. ")
Regarding claim 27, Lim teaches: The method of claim 20, further comprising highlighting detected blobs in the combined image by annotation with graphical or text elements. (Fig. 6A, 622; [0073], "In the scene 600, the participant 132 has advanced well within the threshold distance 510 and the foreground space 515. Based on the above-mentioned discontinuities between the portion 622 and surrounding areas of the depth image 620b, the depth image segmenter 325 identifies the portion 622 as a foreground portion 622 of the depth image 620b.")
Regarding claim 30, Lim teaches: The method of claim 20, wherein the overlaying of the cropped images includes scaling or resizing the cropped images to fit spatially within the combined image. ([0054], "The foreground image generator 345 is configured to resize the foreground portion of the RGB image (for example, by proportional scaling of the foreground portion to a smaller number of pixels) to generate the foreground image, such that a total height of the generated foreground image and/or a total height of the foreground image in the composite image 362 is a second percentage of a total height of the composite image 362, where the second percentage is substantially smaller than the first percentage.")
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 24 and 29 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lim et al. (US Patent Number 10122969 -B1, hereinafter “Lim”) in view of Deng et al. (H. Deng, W. Zhang, E. Mortensen, T. Dietterich and L. Shapiro, "Principal Curvature-Based Region Detector for Object Recognition," 2007 IEEE Conference on Computer Vision and Pattern Recognition, Minneapolis, MN, USA, 2007, pp. 1-8, doi: 10.1109/CVPR.2007.382972., hereinafter “Deng”)
Regarding claim 24, Lim does not teach: The method of claim 20, wherein the blob detection is performed to isolate one or more objects in the images using principal curvature-based region detection.
However, Deng does teach: The method of claim 20, wherein the blob detection is performed to isolate one or more objects in the images using principal curvature-based region detection. (Deng, Introduction, "Conversely, PCBR utilizes line and edge features to construct structural interest regions. Compared to MSER, PCBR differs two important aspects. First, MSER does not analyze regions in scale space, so it does not provide different levels of region abstraction. Second, MSER's intensity-based threshold process cannot overcome local intensity variations within regions. PCBR, however, overcomes this difficulty by focusing on region boundaries rather than the appearance of region interiors.")
At the time the invention was made, it would have been obvious to one of ordinary skill in the art to modify image combination with segmented element through foreground segmenter (Lim) to include segmentation through PCBR (Deng) because such a modification is the result of simple substitution of one known element for another producing a predictable result. More specifically, foreground segmenter and PCBR perform the same general and predictable function, the predictable function being degerming the boundaries and edges of a given structural element. Since each individual element and its function are shown in the prior art, albeit shown in separate references, the difference between the claimed subject matter and the prior art rests not on any individual element or function but in the very combination itself - that is in the substitution of foreground segmenter by replacing it with PCBR. Thus, the simple substitution of one known element for another producing a predictable result renders the claim obvious.
Regarding claim 29, Lim does not teach: The method of claim 20 wherein , the blob detection comprises one or more of: Laplacian of Gaussian (LoG); Difference of Gaussians (DoG); Determinant of Hessian (DoH); Maximally stable extremal regions; or, principal curvature-based region detection. (Deng, Introduction, "Conversely, PCBR utilizes line and edge features to construct structural interest regions. Compared to MSER, PCBR differs two important aspects. First, MSER does not analyze regions in scale space, so it does not provide different levels of region abstraction. Second, MSER's intensity-based threshold process cannot overcome local intensity variations within regions. PCBR, however, overcomes this difficulty by focusing on region boundaries rather than the appearance of region interiors.")
At the time the invention was made, it would have been obvious to one of ordinary skill in the art to modify image combination with segmented element through foreground segmenter (Lim) to include segmentation through PCBR (Deng) because such a modification is the result of simple substitution of one known element for another producing a predictable result. More specifically, foreground segmenter and PCBR perform the same general and predictable function, the predictable function being degerming the boundaries and edges of a given structural element. Since each individual element and its function are shown in the prior art, albeit shown in separate references, the difference between the claimed subject matter and the prior art rests not on any individual element or function but in the very combination itself - that is in the substitution of foreground segmenter by replacing it with PCBR. Thus, the simple substitution of one known element for another producing a predictable result renders the claim obvious.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Jinsu Hwang whose telephone number is (703)756-1370. The examiner can normally be reached Mon - Thu 10am-8am EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matthew Bella can be reached at (571) 272-7778. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JINSU HWANG/Examiner, Art Unit 2667
/MATTHEW C BELLA/Supervisory Patent Examiner, Art Unit 2667
Prosecution Insights