DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant's arguments filed on April 13, 2026 have been fully considered but they are not persuasive.
Applicant argues that the combined teachings of Mueller and Bingham fail to disclose the claim 1 limitations of ''determining a mapping relation between the one or more first pixel feature points and the one or more second pixel feature points” and “determining position information of one or more pixel feature points corresponding to the one or more shadows in a three-dimensional (3D) space based on the one or more first pixel feature points and the one or more second pixel feature points by performing spatial mapping”. The portions of the above-quoted limitations that are in bolded text are newly added to the claim by the instant amendment.
Applicant argues that Bingham does not disclose these limitations. Figs. 4-6 of Bingham are reproduced below for convenience.
PNG
media_image1.png
200
400
media_image1.png
Greyscale
PNG
media_image2.png
200
400
media_image2.png
Greyscale
PNG
media_image3.png
200
400
media_image3.png
Greyscale
Specifically, Applicant argues:
“[f]or example, the technique may include acquiring images of a real scene from two camera input devices simultaneously. For each image, at least two object interest points (IPs) and at least two shadow IPs are identified, and then a correspondence is determined between object IPs and shadow IPs within each image, as shown for example in FIG. 4 of Bingham…However, even assuming arguendo that the shadow IPs of Bingham may be properly aligned with the claimed one or more first pixel feature points and the one or more second pixel feature points, Applicant submits that Bingham fails to disclose determining a mapping relation between the shadow IPs of one image and the shadow IPs of another image, or determining positions of the shadow IPs of Bingham in a 3D space by performing spatial mapping. Instead, Bingham merely appears to disclose determining correspondences between shadow IPs and object IPs within each image.”
The examiner disagrees with Applicant’s characterization of Bingham. Applicant’s characterization seems to be that the interest points (IPs) of the shadow and object of one image are considered separately from the IPs of the shadow and object of the other image and that there is no correspondence between the shadow and object IPs of one image and the shadow and object IPs of the other image. However, that is not true because the method of Bingham relies on their being correspondences between the positions of the IPs in the two images such that the intersection of the lines drawn from the shadow IPs of the two images corresponds to the position of the illumination source in 3-D space. If correspondences between the IPs of the two images were not determined, the method of Bingham would not be able to determine the position of the illumination source in 3-D space.
The example given in Bingham is that, for each image, correspondences of two interest points (IPs) of the shadow with two corresponding IPs of the object are determined (section 3: “[c]orrespondences between geometry IPs and shadow IPs need to be defined. We need at least two correspondences per image but more will offer im-proved accuracy”). Correspondences between the IPs of the two images are then determined so that the positions of the IPs of the two images relative to one another in 3-D space can be determined (section 3: “[a]ssuming we know the angle between the two input images we can position them within a three dimensional scene adjacent to each other and cast lines through the plane of each image”). Because the positions of the IPs of the two images are determined in the same 3-D coordinate system, this indicates that a mapping relation is established between shadow IPs of one of the images and the shadow IPs of the other image. Given all of this information, the intersection of the lines from the shadow IPs of the two images gives the 3-D position of the illumination source (section 3: “[t]he point of intersection is the approximate illuminant position in three dimensional space”).
Therefore, since Bingham determines the 3-D positions of the shadow IPs and object IPs of the first and second images in the same 3-D space, Bingham does disclose the claim 1 limitation of determining a mapping relation between the one or more first pixel feature points (the shadow IPs of the first image in Bingham) and the one or more second pixel feature points (the shadow IPs of the second image in Bingham). Otherwise, the position of the illuminant in 3-D space could not be determined based on the intersection of the lines drawn from the shadow IPs of the two images.
Regarding the claim 1 limitation of “determining position information of one or more pixel feature points corresponding to the one or more shadows in a three-dimensional (3D) space based on the one or more first pixel feature points and the one or more second pixel feature points by performing spatial mapping” (Emphasis Added), Bingham discloses this limitation for the reasons discussed above, i.e., Bingham discloses determining the positions of the shadow IPs of both images in the same 3-D space by using the known angle between the two images to map the shadow and object IPs of the two images from 2-D space to the same 3-D space. This constitutes “performing spatial mapping” to determine position information for the pixel feature points of the shadow IPs in 3-D space.
Regarding Applicant’s arguments regarding Mueller, these arguments are moot because Mueller is no longer relied upon for the same teachings for which it was relied upon in the previous Office Action.
Claim Interpretation
The claims in this application are given their broadest reasonable interpretation (BRI) using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. BRIs for particular claim terms are provided herein. Should Applicant believe that these interpretations are inaccurate, Applicant should point to the portions of the specification that provide a basis for different interpretations.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1, 2, 4, 5, 8, 9, 11, 12 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over an article entitled “Illuminant Condition Matching in Augmented Reality: A Multi-Vision, Interest Point Based Approach”, by Bingham et al., published in 2009 Sixth International Conference on Computer Graphics, Imaging and Visualization (2009, Page(s): 57-61), published on August 11, 2009 (hereinafter referred to as “Bingham”) in view of U.S. Publ. Appl. No. 2005/0180623 A1 of Mueller et al. (hereinafter referred to as “Mueller”) in view of.
Regarding claim 1, Bingham discloses an illuminant estimation method for an electronic device (Section 3, step 5. “5. Locate illuminant in 3D space”. Section 3, “[t]he point of intersection is the approximate illuminant position in three dimensional space. Figure 6 shows the expected results. Once geometric registration has been applied this location will refer to the position of the real illuminant as shown by the sphere (at the 3D intersection point) in the image. See Fig. 6 duplicated above in the Response to Arguments section of this Office Action), the method comprising:
acquiring two image frames comprising a first image frame captured at a first position and a second image frame captured at a second position, wherein a distance between the two image frames is greater than a predetermined distance (Section 3, Step 1: “Acquire Images”. Section 3: “Images of the real scene are obtained from two cam-era input devices simultaneously. The images can be taken from any angle. It is anticipated that the more acute the angle the less accurate the result. This prototype system uses two cameras observing the scene at an angle of 90 degree separation”. The predetermined angular separation between the images constitutes a separation between the two image frames by greater than a predetermined distance);
detecting one or more shadows included in the two image frames (Section 3, step 2: “Identify IPs”; section 3: “Interest points may be detected using one of the techniques discussed in section 2. The Scale Invariant Feature Transform (SIFT) method presented by Lowe [15] is preferred as it makes available additional information that is of use when detecting the correspondence between shadow interest points and object interest points”. Section 2 discusses different techniques for detecting and identifying shadow IPs);
extracting one or more first pixel feature points corresponding to the one or more shadows from the first image frame and one or more second pixel feature points corresponding to the one or more shadows from the second image frame (The shadow IPs are extracted as part of the IP detection step. Figs. 4 and 5, duplicated above in the Response to Arguments section of this Office Action, show the extracted shadow and object IPs. Section 3 also discusses extraction: “[o]nce interest points have been obtained they need to be classified as being associated with either a cast shadow or object geometry. Correspondences between geometry IPs and shadow IPs need to be defined. We need at least two correspondences per image but more will offer im-proved accuracy. Figure 4 shows correspondences between shadow and object IPs”);
determining a mapping relation between the one or more first pixel feature points and the one or more second pixel feature points (Correspondences between the IPs of the two images are determined so that the positions of the IPs of the two images relative to one another in 3-D space can be determined, section 3: “[a]ssuming we know the angle between the two input images we can position them within a three dimensional scene adjacent to each other and cast lines through the plane of each image”. Because the positions of the IPs of the two images in the same 3-D coordinate system are determined, this indicates that a mapping relation is established between shadow IPs of one of the images and the shadow IPs of the other image);
determining position information of one or more pixel feature points corresponding to the one or more shadows in a three-dimensional (3D) space based on the one or more first pixel feature points and the one or more second pixel feature
points by performing spatial mapping (as indicated above, Bingham discloses determining the positions of the shadow IPs of both images in the same 3-D space by using the known angle between the two images to map the shadow and object IPs of the two images from 2-D space to the same 3-D space. This constitutes “performing spatial mapping” to determine position information for the pixel feature points of the shadow IPs in 3-D space.);
acquiring point cloud information about multiple objects corresponding to one or more objects from the first image frame (Bingham does not explicitly disclose that generating the 3-D representation of the objects and their shadows in 3-D space is a point cloud representation); and
determining a position of an illuminant based on the position information of the one or more pixel feature points corresponding to the one or more shadows in the 3D space and the point cloud information corresponding to the one or more objects (section 3 discloses that the position of the illuminant is determined based on the position information of the one or more pixel feature points corresponding to the one or more shadows in the 3D space. Specifically, section 3 discloses that drawing lines from the shadow IPs of the two images through the corresponding object IPs of the two images and beyond to their point of intersection determines the position of the illuminant: “Figure 5 shows the correspondence lines and detected illuminant positions for two images…Assuming we know the angle between the two input images we can position them within a three dimensional scene adjacent to each other and cast lines through the plane of each image. Slight inaccuracies can be expected therefore to ensure these two lines actually intersect an average height value is taken. The point of intersection is the approximate illuminant position in three dimensional space. Figure 6 shows the expected results.” See Fig. 6 duplicated above in the Response to Arguments section of this Office Action).
As indicated above, Bingham does not explicitly disclose that generating the 3-D representation of the objects and their shadows in 3-D space is a point cloud representation. Mueller, in the same field of endeavor, discloses acquiring point cloud information about multiple objects corresponding to one or more objects from one or more image frames as point cloud information that is transformed into a 3-D mesh (para. [0165]).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the present disclosure, to modify the system and method of Bingham based on the teachings of Mueller to convert the 3-D representation of the images shown in Fig. 5 of Bingham into point cloud information and to use the point cloud information in the process of determining the position of the illuminant. A person of ordinary skill would have been motivated to make the modification to achieve higher geometric precision and reduce computational complexity. The modification could have been made by one of ordinary skill in the art before the effective filing data of the present disclosure with a reasonable expectation of success because making the modification merely involves combining prior art elements according to known methods to yield predictable results (utilizing software in the system of Bingham to the convert 3-D coordinates of the objects and shadows in the images into point cloud data and to process the point cloud data).
Regarding claim 2, Bingham discloses that determining of the position of the illuminant comprises:
for each object, emitting a ray from a point farthest from the each object from among all edge points of a corresponding shadow to a highest point of the each object (Fig. 5, duplicated above, shows rays being emitted from IPs on the edge of the shadows farthest away from the object to a highest point of each object. Section 3 describes the line drawing); and
determining an intersection of at least two rays or an intersection of one ray and an illuminant direction predicted by an illumination estimation model as the position of the illuminant (as indicated above in the rejection of claim 1, Bingham discloses that the point of intersection of at least two of the rays is determined to be the position of the illuminant. This is shown in Fig. 6, duplicated above).
Regarding claim 4, Bingham discloses that determining the mapping relation between the one or more first pixel feature points and the one or more second pixel feature points comprises mapping the one or more first pixel feature points corresponding to the one or more shadows back to the first image frame and the one or more second pixel feature points corresponding to the one or more shadows back to the second image frame (Section 3 discusses determining “correspondences” between the positions of the shadow IPs and the corresponding object IPs on a ”per image” basis. Fig. 4 shows the correspondences between the shadow and object IPs of one of the images. Fig. 5 shows the correspondences for two of the images. As indicated above, section 3 discloses that because the angle between the images is known, the correspondences between the positions of the shadow IPs in the shadow IPs in the other image can be determined in the same 3-D space and used during line drawing to determine the position of the illuminant in 3-D space: “[a]ssuming we know the angle between the two input images we can position them within a three dimensional scene adjacent to each other and cast lines through the plane of each image. Slight inaccuracies can be expected
therefore to ensure these two lines actually intersect an average height value is taken. The point of intersection is the approximate illuminant position in three dimensional space. Figure 6 shows the expected results”. Determining these “correspondences” between positions of IPs in Bingham constitutes mapping the one or more first pixel feature points corresponding to the one or more shadows (the shadow IPs of the first image in Bingham) back to the first image frame and the one or more second pixel feature points corresponding to the one or more shadows (the shadow IPs of the second image in Bingham) back to the second image frame).
Regarding claim 5, the BRI for this limitation, based on Fig. 6 and para. [0050] of the present disclosure, is that for images captured by the respective cameras at respective poses, the feature points of the shadows of the two images are traced back through the corresponding feature points of the objects and beyond until they intersect, which intersection corresponds to the position of the illuminant. As indicated above in the rejection of claim 1, section 3 of Bingham describes this same process with reference to Figs. 5 and 6 in which lines are drawn from the shadow IPs of the two images through the corresponding object IPs and beyond until they intersect at the position of the illuminant. Bingham discloses that the images are acquired by two cameras from different angles, which means they have respective poses (Section 3: “[i]mages of the real scene are obtained from two camera input devices simultaneously. The images can be taken from any angle”).
Regarding claim 8, to the extent that claim 8 recites the same limitations that are recited in claim 1, the rejection of claim 1 applies mutatis mutandis to claim 8. The only limitations that are recited in claim 8 that are the memory storing instructions which, when executed by at least one processor, cause the at least one processor to perform the operations recited in claims 1 and 8. Bingham does not explicitly disclose that a processor and memory are used to perform the process, but does describe known algorithms (e.g., the SIFT algorithm) that can be used to perform the process in software executed by some type of processor. Therefore, some type of memory and processor are used in the system and process of Bingham, although not explicitly mentioned.
Mueller discloses processors 123 and 130 for performing the process described therein by executing instructions stored in a non-transitory computer-readable medium RAM 204 and ROM 206.
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the present disclosure, to modify the system and method of Bingham based on the teachings of Mueller to use a processor to execute instructions stored in a memory as taught by Mueller. A person of ordinary skill would have been motivated to make the modification to perform the process of Bingham in software executed by a suitable process rather than solely in hardware to avoid the cost and complexity of designing and manufacturing custom hardware for this purpose. The modification could have been made by one of ordinary skill in the art before the effective filing data of the present disclosure with a reasonable expectation of success because making the modification merely involves combining prior art elements according to known methods to yield predictable results (utilizing software and a processor in the system of Bingham).
Regarding claim 9, the rejection of claim 2 applies mutatis mutandis to claim 9.
Regarding claim 11, the rejection of claim 4 applies mutatis mutandis to claim 11.
Regarding claim 12, the rejection of claim 5 applies mutatis mutandis to claim 12.
Regarding claim 15, to the extent that claim 15 recites the same limitations that are recited in claim 1, the rejection of claim 1 applies mutatis mutandis to claim 15. The only limitations that are recited in claim 15 that are not also recited in claim 1 are the non-transitory computer-readable storage medium configured to store instructions which, when executed by at least one processor, cause the at least one processor to perform the operations recited in claims 1 and 15. Bingham does not explicitly disclose that a processor and memory are used to perform the process, but does describe known algorithms (e.g., the SIFT algorithm) that can be used to perform the process in software executed by some type of processor. Therefore, some type of computer-readable medium and processor are used in the system and process of Bingham, although not explicitly mentioned.
Mueller discloses processors 123 and 130 for performing the process described therein by executing instructions stored in a non-transitory computer-readable medium RAM 204 and ROM 206.
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the present disclosure, to modify the system and method of Bingham based on the teachings of Mueller to use a processor to execute instructions stored in a computer-readable medium as taught by Mueller. A person of ordinary skill would have been motivated to make the modification to perform the process of Bingham in software executed by a suitable process rather than solely in hardware to avoid the cost and complexity of designing and manufacturing custom hardware for this purpose. The modification could have been made by one of ordinary skill in the art before the effective filing data of the present disclosure with a reasonable expectation of success because making the modification merely involves combining prior art elements according to known methods to yield predictable results (utilizing software and a processor in the system of Bingham).
Claims 3, 6, 10 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Bingham in view of Mueller as applied to claims 1, 2, 4, 5, 8, 9, 11, 12 and 15 and further in view of U.S. Publ. Appl. No. 2014/0341464 A1 of Fan et al. (hereinafter referred to as “Fan”).
Regarding claim 3, the combined teachings of Bingham and Mueller do not explicitly teach converting the two image frames into gray images and obtaining shadows included in the gray images. The BRI for the term “gray images” is that it means grayscale images, based on para. [0046] of the present disclosure.
Fan, in the same field of endeavor, discloses a shadow detection method and device that converts images captured by a camera into grayscale images and obtains shadows included in the gray images (Fig. 2, para. [0029]-[0030]).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the present disclosure, to modify the system and method of Bingham to acquire grayscale images or to convert images acquired during image acquisition step 1 of section 3 into grayscale images that include the shadows and to perform the operations recited in claim 1 on the grayscale images to obtain the shadows as taught by Fan. A person of ordinary skill would have been motivated to make the modification based on a determination that better results are obtained when processing is performed on grayscale images rather than color images since it is well known in the art that either format can be used. The modification could have been made by one of ordinary skill in the art before the effective filing data of the present disclosure with a reasonable expectation of success because making the modification merely involves combining prior art elements according to known methods (modifying or augmenting the image acquisition and processing system of Bingham to acquire or generate grayscale images) to yield predictable results.
Regarding claim 6, the BRI for classifying point clouds belonging to a same object, from among all point clouds, to one category, and classifying point clouds belonging to different objects to different categories is that it means processing the point cloud data to associate points in the point cloud data with respective objects contained in the image frames such that each point cloud is classified as representing a respective object and can be used to distinguish the objects from one another. The BRI is based on paras. [0052]-[0053] of the present disclosure.
The combined teachings of Bingham and Mueller do not explicitly teach this limitation. However, Fan teaches classifying point clouds belonging to a same object, from among all point clouds, to one category, and classifying point clouds belonging to different objects to different categories, where the classified point clouds are identified by index numbers and are processed to identify shadows (paras. [0014]-[0015] and [0049]).
It would have been obvious to a person It would have been obvious to one of ordinary skill in the art, before the effective filing date of the present disclosure, to further modify the system of Bingham as modified based on the teachings of Mueller to distinguish between the point clouds associated with the respective objects by using the classification techniques taught by Fan in the system shown in Fig. 1 of Bingham. A person of ordinary skill would have been motivated to make the modification to take advantage of the well-known benefits of using neural networks to perform clustering and classification to perform object detection and recognition. The modification could have been made by one of ordinary skill in the art before the effective filing data of the present disclosure with a reasonable expectation of success because making the modification merely involves combining prior art elements according to known methods (modifying or augmenting software to be executed by the system of Bingham to distinguish between point clouds associated with respective objects) to yield predictable results.
Regarding claim 10, the rejection of claim 3 applies mutatis mutandis to claim 10.
Regarding claim 13, the rejection of claim 6 applies mutatis mutandis to claim 13.
Claims 7 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Bingham in view of Mueller as applied to claims 1, 2, 4,5, 8, 9, 11, 12 and 15 and further in view of U.S. Publ. Appl. No. 2021/0370968 A1 of Xiao et al. (hereinafter referred to as “Xiao”).
Regarding claim 7, the BRI for determining the corresponding shadows based on a distance between the point cloud of each shadow and the point cloud of each object is based on para. [0055] of the present disclosure. The BRI for this limitation is that a distance between point clouds of the shadows and the point clouds of the respective objects that cast the respective shadows is used to match the point clouds with the corresponding objects.
As indicated above, Bingham does not explicitly disclose that the 3-D information is point cloud information. Bingham discloses determining a correspondence between the IPs of the shadows and the IPs of respective objects that casted the shadows, but does not explicitly disclose using point cloud data as part of the process of determining the correspondence.
Xiao, in the same field of endeavor, discloses a system that identifies point clouds in successive image frames as corresponding to shadows (para. [0171]) and point clouds corresponding to objects (para. [0167]). In addition, Xiao discloses determining a correspondence between point clouds based on a distance between the point clouds by minimizing a sum of squared differences between coordinates of points of the point clouds (para. [0088]).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the present disclosure, to further modify the system and method of Bingham as modified based on the teachings of Mueller to determine correspondences between point clouds representing shadows and point clouds representing the respective objects based on a distance between the point clouds as taught by Xiao. A person of ordinary skill would have been motivated to make the modification to take advantage of the well-known benefits associated with using point cloud data to represent and identify objects. The modification could have been made by one of ordinary skill in the art before the effective filing data of the present disclosure with a reasonable expectation of success because making the modification merely involves combining prior art elements according to known methods (modifying or augmenting software used in the system of Bingham) to yield predictable results.
Regarding claim 14, the rejection of claim 7 applies mutatis mutandis to claim 14.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DANIEL J SANTOS whose telephone number is (571)272-2867. The examiner can normally be reached M-F 9-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matt Bella can be reached on (571)272-7778. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DANIEL J. SANTOS/Examiner, Art Unit 2667
/MATTHEW C BELLA/Supervisory Patent Examiner, Art Unit 2667
Prosecution Insights