DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant's arguments filed on November 24, 2025 have been fully considered but they are not persuasive. Additionally, Applicant’s amendments have necessitated a new ground of rejection.
Applicant provided a “Statement of Substance of Interview” in the response to the nonfinal Office Action in which Applicant states that “[d]uring the interview, the Examiner indicated that amendments similar to those presented herein would most likely be sufficient to overcome the outstanding rejections.” The Interview Summary Record prepared by the examiner states: “[t]he examiner and Attorney Negley discussed proposed amendments to the claims and whether the claims as amended in the proposal would overcome the rejection of the independent claims under 35 U.S.C. 103 over Wang et al. in view of Mueller et al. It should be noted that in the Interview Summary Record, the examiner indicated that the “proposed amendment would require an additional search and/or additional consideration of the Mueller reference.”
The examiner has further considered the Wang reference and the Mueller reference and has determined that the present amendment to the independent claims does overcome the rejection of claims 1, 2, 8, 9 and 15 under 35 U.S.C. 103 as being unpatentable over Wang in view of Mueller. However, a new ground of rejection of these claims is set forth below.
Regarding the Mueller reference, Applicant argues that in Mueller the object remains stationary and the detector moves around the object, but that “this is for the purpose of 3D modeling of an object, not for light source estimation or shadow detection. Specifically, the example disclosed in Mueller for calibrating the reference-coordinate space location of incident lighting (See, e.g., Mueller, paragraph [0164]) uses a cone-shaped three-dimensional calibration object with a checkerboard pattern, and does not disclose using two image frames.”
The examiner disagrees. In Mueller, multiple image frames of the calibration objects (e.g., the planar checkerboard patterns shown in Fig. 13 or the cone-shaped calibration objects shown in Fig. 19) and of the subject 3D object are acquired because the calibration objects are located in a calibration ring on the same support base on which the subject object is located such that the subject object is surrounded by the calibration objects (Figs. 13, 14 and 19). The calibration ring surrounds the subject object so that at least two of the calibration objects will be within the FOV of the image detector(s) in each frame and not occluded by the subject object (para. [0134]).
In Mueller, multiple image frames are captured of the subject object and of the calibration objects in one of multiple ways. In one embodiment of Mueller, multiple image capture devices positioned at predetermined locations about the subject object (Fig. 2, paras. [0132]-[0139]) are used such that each image capture device captures an image frame from a different position. In other embodiments, Mueller discloses using a single image capture device and generating relative motion between the image capture device and the subject object by moving the image capture device about the object while keeping the subject object stationary or by rotating the platform on which the subject object is located while keeping the image capture device stationary (Figs. 1 - 1B, para. [0048]-[0055]).
In all of these scenarios of Mueller, multiple image frames are captured of the subject object and of the calibrations objects since they are positioned on the same platform, as shown in Figs. 13 and 19. Triangulation is then performed using feature points of the calibration objects and feature points of the shadows cast by the calibration objects (or of shadows cast by the same calibration object in two different known positions relative to the image capture device, para. [0134]) to determine the location of the illuminant (para. [0136]: “[t]he integration of 3D calibration objects with the calibration ring provides the capacity to determine the position and incident angel of light sources”; see also para. [0164] describing using triangulation to determine the location of the light source 93 shown in Fig. 19: “[t]he shadow that each well-known calibration cone casts on the platform determines a line 92 from the tip of each shadow 94, through the tip of the corresponding cone 95, to the light source 93. Triangulation, intersecting two or more cone-shadow lines 92 locates the light source 93 for this purpose.”).
Therefore, Mueller does disclose using two or more image frames to determine the position of the illuminant based on the positions of feature points of shadows and of objects in the image frames. However, Mueller does not explicitly disclose determining the position of the illuminant based on the positions of first and second sets of feature points of shadows in first and second image frames, respectively, that are separated by a predetermined distance.
Applicant also argues that there is no motivation for combining the teachings of Wang and Mueller. The examiner disagrees. Both references are directed to using feature points of shadows cast by objects and feature points of the objects to determine the position of the source of illumination. One of ordinary skill in the art, before the effective filing date of the present disclosure, would have been motivated to combine the teachings of Wang and Mueller to achieve higher accuracy in determining the locations of light sources in order to more accurately reconstruct shadows in scenes and make them appear more realistic. Nevertheless, this issue is moot in view of the new grounds of rejection.
Applicant also argues that several newly added limitations that have been added to independent claims 1, 8 and 15 by the present amendment are not taught by the combined teachings of Wang and Mueller. These newly added limitations are discussed below in the rejections of the claims.
Claim Interpretation
The claims in this application are given their broadest reasonable interpretation (BRI) using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. BRIs for particular claim terms are provided herein. Should Applicant believe that these interpretations are inaccurate, Applicant should point to the portions of the specification that provide a basis for different interpretations.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1, 2, 8, 9 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over U.S. Publ. Appl. No. 2005/0180623 A1 of Mueller et al. (hereinafter referred to as “Mueller”) in view of an article entitled “Illuminant Condition Matching in Augmented Reality: A Multi-Vision, Interest Point Based Approach”, by Bingham et al., published in 2009 Sixth International Conference on Computer Graphics, Imaging and Visualization (2009, Page(s): 57-61), published on August 11, 2009 (hereinafter referred to as “Bingham”).
Regarding claim 1, Mueller discloses an illuminant estimation method for an electronic device (para. [0164] discusses using triangulation to estimate the position of the light source) the method comprising:
acquiring two image frames comprising a first image frame captured at a first position and a second image frame captured at a second position, wherein a distance between the two image frames is greater than a predetermined distance (multiple image frames of the calibration objects (e.g., the cone-shaped calibration objects shown in Fig. 19) and of the subject 3D object are captured because the calibration objects are located in a calibration ring on the same support base on which the subject 3D object 32 is located (Figs. 13, 14 and 19). Multiple image frames are captured of the subject object and of the calibration objects in one of multiple ways. In one embodiment of Mueller, multiple image capture devices positioned at predetermined locations about the subject object (Fig. 2, paras. [0132]-[0139]) are used such that each image capture device captures an image frame from a different position. In other embodiments, Mueller discloses using a single image capture device and generating relative motion between the image capture device and the subject object by moving the image capture device about the object while keeping the subject object stationary or by rotating the platform on which the subject object is located while keeping the image capture device stationary (Figs. 1 - 1B, para. [0048]-[0055]));
detecting one or more shadows included in the two image frames (in each acquired image frame, the tips of the shadows 94 cast by each of at least two calibration objects 91 are detected, Fig. 19, para. [0164]);
extracting one or more first pixel feature points corresponding to the one or more shadows from the first image frame and one or more second pixel feature points corresponding to the one or more shadows from the second image frame (Fig. 19, para. [0164], pixel feature points corresponding to the tips 94 of the shadows cast by at least two calibration objects 91 are extracted from one image frame and pixel feature points corresponding to the tips 94 of the shadows cast by at least two calibration objects 91 are extracted from another image frame);
determining position information of one or more pixel feature points corresponding to the one or more shadows in a three-dimensional (3D) space based on the one or more first pixel feature points and the one or more second pixel feature
points (para. [0165], the surface geometry, which includes the 3D calibration cones 91 and the shadows 94 they cast, is defined in 3D space, and therefore the pixel feature points corresponding the shadow tips 94 are also defined in 3D space);
acquiring point cloud information about multiple objects corresponding to one or more objects from the first image frame (para. [0165], the surface geometry, which includes the 3D calibration cones 91, is acquired as point cloud information that is transformed into a 3D mesh); and
determining a position of an illuminant based on the position information of the one or more pixel feature points corresponding to the one or more shadows in the 3D space and the point cloud information corresponding to the one or more objects (paras. [0164]-[0165], Fig. 19, the position of the light source 93 is determined based on the 3D position information of the tips 94 of the shadows and based on the point cloud information corresponding to the tips 95 of the 3D calibration cones 91; Mueller does not explicitly disclose that the position information of the one or more pixel feature points corresponding to the shadows that is used to determine the position of the illuminant is determined from first and second pixel feature points extracted from first and second image frames, respectively).
Bingham, in the same field of endeavor, discloses extracting first and second pixel feature points corresponding to interest points (IPs) of shadows from first and second image frames that are a predetermined distance apart based on the positions of first and second cameras that are used to simultaneously acquire the first and second image frames, respectively, extracting feature points of an object that casted the shadows from the first and second image frames, determining a correspondence between the pixel feature points of the shadows and the pixel feature points of the object for the two image frames in 2D space, combining the results to obtain 3D coordinates, and then determining the position of the illuminant using all of this positional information (See section 3 and Figs. 4-6: “[o]nce interest points have been obtained they need to be classified as being associated with either a cast shadow or object geometry. Correspondences between geometry IPs and shadow IPs need to be defined. We need at least two correspondences per image but more will offer im-proved accuracy. Figure 4 shows correspondences between shadow and object IPs. If we draw a line from the shadow interest point towards the corresponding object interest point and beyond we can obtain the approximate location of the illuminant in 2D co-ordinate space. Intersections between two or more lines indicate a potential illuminant position.” See Fig. 6 duplicated below showing the intersection of the lines indicating the position of the illuminant).
PNG
media_image1.png
200
400
media_image1.png
Greyscale
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the present disclosure, to modify the system and method of Mueller based on the teachings of Bingham to use positional information about extracted pixel feature points corresponding to shadows extracted from at least first and second image frames in combination with positional information about pixel feature points corresponding to the object that casted the shadows from the first and second image frames to determine the illuminant position as taught by Bingham. A person of ordinary skill would have been motivated to make the modification obtain a more accurate estimate for the illuminant position by averaging the results and discarding anomalous results as taught by Bingham (section 3: “[i]ntersections between two or more lines indicate a potential illuminant position. Any anomalous results are discarded and the average intersection point is recorded as the 2D illuminant position”). The modification could have been made by one of ordinary skill in the art before the effective filing data of the present disclosure with a reasonable expectation of success because making the modification merely involves combining prior art elements according to known methods to yield predictable results (modifying or augmenting the software and/or firmware of the image processor 130 to perform the processing across multiple image frames).
Regarding claim 2, the BRI for this limitation, which is based on para. [0063] of the specification and Fig. 9 of the drawings of the present disclosure, is that it means using triangulation with at least two rays projected from points on the edges of the shadows that are farthest away from a highest point on the corresponding objects to a point where the rays intersect, the intersection corresponding to the position of the illuminant.
Mueller discloses using triangulation with at least two rays projected from points on the edges of the shadows (Fig. 19, para. [0164], the shadow tips 94) that are farthest away from a highest point on the corresponding objects (Fig. 19, para. [0164], the tips 95 of the calibration cones 91) to a point where the rays intersect (Fig. 19, the light source 93), the intersection corresponding to the position of the illuminant (the rays shown in Fig. 19 intersect at the light source 93).
Regarding claim 8, to the extent that claim 8 recites the same limitations that are recited in claim 1, the rejection of claim 1 applies mutatis mutandis to claim 8. The only limitations that are recited in claim 8 that are not also recited in claim 1 are the image acquisition unit, the shadow detection unit, the object distinguishing unit, the shadow matching unit, and the illuminant estimation unit. As indicated above, the BRI for all of these terms is one or more processors and one or more memory devices that store instructions, which, when executed by the one or more processors, perform the respective operations recited in claim 8, which are also recited in claim 1. Figs. 1 and 2 of Bingham illustrates processors 120 and 130 and memory 204/206 for performing the operations recited in claims 1 and 8 (paras. [0051]- [0059]).
Regarding claim 9, the rejection of claim 2 applies mutatis mutandis to claim 9.
Regarding claim 15, to the extent that claim 15 recites the same limitations that are recited in claim 1, the rejection of claim 1 applies mutatis mutandis to claim 15. The only limitations that are recited in claim 15 that are not also recited in claim 1 are the non-transitory computer-readable storage medium configured to store instructions which, when executed by at least one processor, cause the at least one processor to perform the operations recited in claims 1 and 15. As indicated above, Bingham discloses processors 123 and 130 for performing by executing instructions stored in a non-transitory computer-readable medium RAM 204 and ROM 206.
Claims 3, 6, 10 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Mueller in view of Bingham as applied to claims 1, 2, 8, 9 and 15 and further in view of U.S. Publ. Appl. No. 2014/0341464 A1 of Fan et al. (hereinafter referred to as “Fan”).
Regarding claim 3, the combined teachings of Mueller and Bingham do not explicitly teach converting the two image frames into gray images and obtaining shadows included in the gray images. The BRI for the term “gray images” is that it means grayscale images, based on para. [0046] of the present disclosure.
Fan, in the same field of endeavor, discloses a shadow detection method and device that converts images captured by a camera into grayscale images and obtains shadows included in the gray images (Fig. 2, para. [0029]-[0030]). It would have been obvious to one of ordinary skill in the art, before the effective filing date of the present disclosure, to modify the image acquisition and processing system of Mueller (Fig. 1, image detector 110 and image input processing system 120) to acquire grayscale images or to convert images acquired by the image detector 110 into grayscale images that include the shadows and to perform the operations recited in claim 1 on the grayscale images to obtain the shadows as taught by Fan. A person of ordinary skill would have been motivated to make the modification based on a determination that better results are obtained when processing is performed on grayscale images rather than color images since it is well known in the art that either format can be used. The modification could have been made by one of ordinary skill in the art before the effective filing data of the present disclosure with a reasonable expectation of success because making the modification merely involves combining prior art elements according to known methods (modifying or augmenting the image acquisition and processing system 110, 120 shown in Fig. 1 of Mueller) to yield predictable results.
Regarding claim 6, the BRI for classifying point clouds belonging to a same object, from among all point clouds, to one category, and classifying point clouds belonging to different objects to different categories is that it means processing the point cloud data to associate points in the point cloud data with respective objects contained in the image frames such that each point cloud is classified as representing a respective object and can be used to distinguish the objects from one another. The BRI is based on paras. [0052]-[0053] of the present disclosure.
The combined teachings of Mueller and Bingham do not explicitly teach this limitation. However, Fan teaches classifying point clouds belonging to a same object, from among all point clouds, to one category, and classifying point clouds belonging to different objects to different categories, where the classified point clouds are identified by index numbers and are processed to identify shadows (paras. [0014]-[0015] and [0049]).
It would have been obvious to a person It would have been obvious to one of ordinary skill in the art, before the effective filing date of the present disclosure, to further modify the system of Mueller as modified based on the teachings of Bingham to distinguish between the point clouds associated with the respective objects by using the classification techniques taught by Fan in the system shown in Fig. 1 of Bingham. A person of ordinary skill would have been motivated to make the modification to take advantage of the well-known benefits of using neural networks to perform clustering and classification to perform object detection and recognition. The modification could have been made by one of ordinary skill in the art before the effective filing data of the present disclosure with a reasonable expectation of success because making the modification merely involves combining prior art elements according to known methods (modifying or augmenting software to be executed by the system shown in Fig. 1 of Bingham) to yield predictable results.
Regarding claim 10, the rejection of claim 3 applies mutatis mutandis to claim 10.
Regarding claim 13, the rejection of claim 6 applies mutatis mutandis to claim 13
Claims 7 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Mueller in view of Bingham as applied to claims 1, 2, 8, 9 and 15 and further in view of U.S. Publ. Appl. No. 2021/0370968 A1 of Xiao et al. (hereinafter referred to as “Xiao”).
Regarding claim 7, the BRI for determining the corresponding shadows based on a distance between the point cloud of each shadow and the point cloud of each object is based on para. [0055] of the present disclosure. The BRI for this limitation is that a distance between point clouds of the shadows and the point clouds of the respective objects that cast the respective shadows is used to match the point clouds with the corresponding objects.
As indicated above, Mueller discloses obtaining point cloud data for objects and shadows (paras. [0164]-[0165]) and using the detected objects and the respective shadows to estimate the position of the light source (para. [0164], Fig. 19). As indicated above, Fan discloses converting captured images into point clouds and performing classification to classify point clouds using index numbers to identify point clouds corresponding to shadows (paras. [0014]-[0015] and [0049]). As indicated above, Bingham discloses determining a correspondence between the IPs of the shadows and the IPs of respective objects that casted the shadows, but does not explicitly disclose using point cloud data as part of the process of determining the correspondence.
Xiao, in the same field of endeavor, discloses a system that identifies point clouds in successive image frames as corresponding to shadows (para. [0171]) and point clouds corresponding to objects (para. [0167]). In addition, Xiao discloses determining a correspondence between point clouds based on a distance between the point clouds by minimizing a sum of squared differences between coordinates of points of the point clouds (para. [0088]).
It would have been obvious to a person It would have been obvious to one of ordinary skill in the art, before the effective filing date of the present disclosure, to further modify the system and method of Mueller as modified above based on the teachings of Bingham to determine correspondences between point clouds representing shadows and point clouds representing the respective objects based on a distance between the point clouds as taught by Xiao. A person of ordinary skill would have been motivated to make the modification to take advantage of the well-known benefits associated with using point cloud data to represent and identify objects. The modification could have been made by one of ordinary skill in the art before the effective filing data of the present disclosure with a reasonable expectation of success because making the modification merely involves combining prior art elements according to known methods (modifying or augmenting software to be executed by the image processor 130 of Mueller) to yield predictable results.
Allowable Subject Matter
Claims 4, 5, 11 and 12 would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter. Regarding claims 4 and 11, these claims recite that determining the position information of the one or more pixel feature points corresponding to the one or more shadows in the 3D space comprises mapping the one or more pixel feature points corresponding to the one or more shadows back to the first image frame and the one or more second pixel feature points corresponding to the one or more shadows back to the second image frame, for the two image frames, determining a mapping relation between the one or more first pixel feature points and the one or more second pixel feature points, and obtaining the position information of the one or more pixel feature points corresponding to the one or more shadows in the 3D space based on spatial mapping.
None of the prior art teaches or suggests these limitations in combination with the other limitations of the respective base claims. Claims 5 and 12 are allowable due to their dependencies from claims 4 and 11, respectively.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
U.S. Pat. No. 9,600,927 B1 discloses a system and method for determining the shape of an object based on shadows cast by the object based on acquiring multiple image frames. The method includes receiving a plurality of image frames of an object casting a shadow. Each image frame may include a shadow cast by the object as the object is illuminated by a light source that moves over a plurality of positions. The method further includes determining, respectively for each image of the plurality of images, a 2D silhouette of the object and a respective position of the light source relative to the object. A 3D object data model of the object may be generated based on the 2D silhouette of the object and the respective position of the light source relative to the object for each image frame of the plurality of image frames.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DANIEL J SANTOS whose telephone number is (571)272-2867. The examiner can normally be reached M-F 9-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matt Bella can be reached on (571)272-7778. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DANIEL J. SANTOS/Examiner, Art Unit 2667
/MATTHEW C BELLA/Supervisory Patent Examiner, Art Unit 2667