DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 1-4, 6, 10 and 13-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Tang et al. (US 2019/0213742)(“Tang”), and further in view of Pettegrew et al. (US 2016/0105609)(“Pettegrew”).
With regards to claim 1, 19-20, Tang discloses a methods for devices that process image data for creating video abstraction based on image data captured by an imaging device borne by a movable object (Abstract, [0004]), a system for controlling movement of an object (Abstract, [0004]), a non-transitory medium storing program instructions for controlling movement of an object [0042], (Abstract, [0004]) comprising:
at least one processor executing a code [0108] for:
accessing a current image of a surface [0063][0040]; image subject (e.g., a target or a remote object, “…tracking data (e.g., a graphical tracking indicator applied to a representation of a target)…”) relative to an object ([0024][0063]; movable object 102) at a current location [0044][0058] ([0044]; “…movable object 102 and/or control unit 104 use sensing data generated by sensors of sensing system 122 to determine information such as a position of movable object 102, an orientation of movable object 102, movement characteristics of movable object 102 (e.g., angular velocity, angular acceleration, translational velocity, translational acceleration and/or direction of motion along one or more axes), proximity of movable object 102 to potential obstacles, weather conditions, locations of geographical features and/or locations of manmade structures.”);
wherein an imaging sensor (FIG. 2A; image sensor 220)([0055]; image sensor 308) is set to capture the current image depicting previously captured image of the surface when the object was at a previous location ([0063]; “…the electronic device acquires (410) a plurality of image frames. The plurality of image frames are captured by the imaging device 216 borne on the movable object 102 when the movable object 102 moves along a path or hovers at a certain height.”)([0066]; “An identified keyframe may correspond to a positional change of the movable object 102 and/or a change of an object in the environment.”;
computing a transformation between the current image and the previous image by considering parameters of the imaging sensor [0070][0081]; and
feeding the current location into a controller for controlling movement of the object [0022][0043]([0044]);
Tang teaches of capturing a plurality of image frames relating to a positional change of the movable object 102 and/or a change of an object in the environment [0066]. Tang further teaches of using computational image stitching steps to generate a panoramic image based on captured images [0050]. Also, Tang teaches “…the electronic device identifies (704) a plurality of image frames of interest (e.g., image frames of interest 506 of FIG. 5) from a sequence of image frames (e.g., image frames 502 of FIG. 5) captured by the imaging device 216.”Finally, Tang teaches that the movable object 102 and/or control unit 104 uses sensor data to determine information such as a position of movable object 102, direction of motion along one or more axes), and proximity of movable object 102 to potential obstacles [0044].
Tang do not specifically disclose;
capturing the current image depicting an overlap with a previously captured image of the surface when the object was at a previous location;
sequentially stitching the current image to the overlap of the previously captured image to create a panorama;
computing the current location of the object relative to a reference location defined for a previously captured baseline image by applying the transformation to the previously computed location determined for the previous image.
In the same field of endeavor, Pettegrew discloses systems and methods to acquire individual frames under a variety of different operating parameters, piece together strips of frames from the individual frames to create a panoramic image (Abstract)(Also; The panoramic view imaging system (PVIS) can be utilized in a drone to move the drone to a position where it has a line of sight to a blind spot. [0032][0064], comprising;
capturing a current images depicting an overlap with a previously captured image of the surface when the object was at a previous location [0065][0079][0092][0093];
sequentially stitching the current image to the overlap of the previously captured image to create a panorama [0078][0079];
computing the current location of the object relative ([0114]; “Method 2300 includes, at 2310, establishing a range finder at a location. The location has an absolute location or a relative location.”) to a reference location ([0088]; “Camera 2100 includes a range finding apparatus 2180 that provides range information for locations identified by the positioning apparatus 2170.”) ([0114]; “…method 2300 may use the range finder as a reference point…”) ([0114]; “Method 2300 also includes, at 2320, defining a set of points to scan with the range finder. The set of points to scan may be defined by an azimuth range, an azimuth resolution, an elevation range, and an elevation resolution.”)
defined for a previously captured baseline image by applying a transformation [0059]([0126]; right-triangle calculation) to the previously computed location determined for the previous image ([0034]([0045]; “System 100 may selectively adapt the range information associated with the plurality of locations in the panoramic image based, at least in part, on the range information for the third, bisecting point.”)[0048][0063].
In view of Pettegrew, it would have been obvious to one of ordinary skill within the art before the effective filing data of the claimed invention to modify the method of Tang, to employ the methodology of obtaining current images of a target (relative to the movable object) that are overlapping with previously captured target images (wherein the images also defines different locations of the target) at various locations and stitch together the overlapping images to create a panoramic image. The panoramic image is subsequently goes through a transformation process and to provide correlation to a reference location. The method is to utilize Pettegrew’ s method to help control the movable object moving to different locations and to help avoid and/or locate targets and obstacles.
With regards to claim 2, Tang, in view of Pettegrew, discloses the computer implemented method of claim 1, further comprising registering the current image to the previously captured image at an overlapping portion of the current image and the previously captured image, wherein the stitching is performed at the overlapping registered portion. (Tang; [0076][0099] in view of the rejection of claim 1)
With regards to claim 3, Tang, in view of Pettegrew, discloses the computer implemented method of claim 1, further comprising identifying multiple keypoint features of the current image (Pettegrew; [0044][0055][0077][0082])(Tang; [0044]), identifying multiple keypoint features of the previous image (Pettigrew; [0038][0040]), mapping the two sets of keypoint features (Tang; [0058])(Pettegrew; [0044][0047][0063][0083]), and stitching the current image to the overlap of the previously captured image according to the mapped two sets of keypoint features (Pettegrew; [0083][0108]).
With regards to claim 4, Tang, in view of Pettegrew, discloses the computer implemented method of claim 1, at least one of:
the object comprises a car and controlling movement comprises automatically parking the car in a parking spot (Tang; [0017]; a self-driving car (i.e., a self-driving car (i.e., an autonomous car, a driverless car)ar, a driverless car)), and
the object comprises a vehicle (Tang; [0006][0017][0018]; UAV)(Pettigrew; [0064][0094]; drone) and the controller automatically generates audio and/or video and/or text instructions instructing a driver how to maneuver the vehicle (Tang; [0028]).
With regards to claim 6, Tang, in view of Pettegrew, discloses the computer implemented method of claim 1, wherein the current location is computed according to a conversion between pixels of the stitched images and physical distances based on parameters of the imaging sensor. (Tang; [0040][0066][0077])(Pettigrew; [0034])
With regards to claim 10, Tang, in view of Pettegrew, discloses the computer implemented method of claim 1, wherein the transformation comprises a projective transformation. (Pettigrew; [0059])
With regards to claim 13, Tang, in view of Pettegrew, discloses the computer implemented method of claim 1, wherein the transformation is computed by considering parameters of the imaging sensor including orientation (Tang; [0023][0041][0071]) and/or distance from the surface (Tang; [0040][0041])
With regards to claim 14, Tang, in view of Pettegrew, discloses the computer implemented method of claim 1, wherein the transformation is represented as a vector indicating a direction and displacement amount of the current image relative to the previous image. (Tang; [0077]; optical flow vectors)
With regards to claim 15, Tang, in view of Pettegrew, discloses the computer implemented method of claim 1, wherein the transformation indicates a shift and/or offset of the current image relative to the previous image in a two dimensional plane relative to the surface. (Pettigrew; [0082])
With regards to claim 16, Tang, in view of Pettegrew, discloses the computer implemented method of claim 1, wherein the current location is computed by adding the transformation comprising a shift and/or an offset in a 2D plane to a previously computed location identified for the previous image. (Pettigrew; [0082])
With regards to claim 17, Tang, in view of Pettegrew, discloses the computer implemented method of claim 1, wherein the imaging sensor is installed on the object and set for capturing images of the surface over which the object is moving. (Tang; [0026][0030][0040])
With regards to claim 18, Tang, in view of Pettegrew, discloses the computer implemented method of claim 1, wherein the accessing, the sequentially switching, the computing the transformation, the computing the current location and the feeding, are iterated dynamically while the object is in motion over a plurality of time intervals using subsequently obtained new current images using preceding obtained images. (Pettigrew; [0113][0124])
Allowable Subject Matter
Claims 5, 7-9 and 11-12 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter:
With regards to claim 5, Tang, in view of Pettegrew, discloses the computer implemented method of claim 1, further comprising activating an electromagnetic illumination source at a selected illumination pattern for generating electromagnetic illumination for providing visibility of the surface depicted in images, wherein the images are captured by an imaging sensor at the electromagnetic spectrum of the electromagnetic illumination, wherein the electromagnetic illumination source generates electromagnetic illumination at the short wave infrared (SWIR) range, and the imaging sensor comprises a SWIR sensor, wherein the SWIR range comprises a solar blind range.
With regards to claim 7, Tang, in view of Pettegrew, do not disclose the computer implemented method of claim 1, further comprising computing a difference between the current image and the previous image at the overlapping region, and indicating a correct stitching when the difference is below a threshold indicating likelihood of a match of the overlapping regions of the current image and the previous image, wherein the threshold is selected to account for temporary objects occluding the surface.
Claims 8-9 are objected due to being dependent on objected base claim 7.
With regards to claim 11, Tang, in view of Pettegrew, do not disclose the computer implemented method of claim 1, further comprising normalizing light intensity of the current image and the previous image on a local basis, wherein the stitching is performed for the normalized current image and the normalized previous image.
Claim 12 is objected due to being dependent on objected base claim 11.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Sanchez-Monge et al. (US 2021/0049738)
Sonn et al. (US 2019/0056498)
Kurtz et al. (US 2023/0090281)
Reiffel (US 2012/0274775)
George et al. (US 2022/0020114)
Chew (US 2013/0329052)
Earhart et al. (US 9,041,915)
Any inquiry concerning this communication or earlier communications from the examiner should be directed to HUGH H MAUPIN whose telephone number is (571)270-1495. The examiner can normally be reached M-F 7:30 - 5:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Uzma Alam can be reached at 571-272-3995. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/HUGH MAUPIN/ Primary Examiner, Art Unit 2884
