Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 4-5, 7 are rejected under 35 U.S.C. 103 as being unpatentable over
Schaarschmidt et al (US20180033212) in view of Nussbaum et al (US10832476),
Tech_Drone_Media (“How to do 3D Mapping with Pix4D – Part 1”, Jul 10, 2017).
Regarding Claim 1. Schaarschmidt teaches A method for obtaining pictures for use in digital replica of a vehicle (Schaarschmidt, abstract, the invention describes a method for generating an active visual product configuration based on interactive digital 2D or 3D objects displayed on an operating device. Processing options and degrees of freedom and/or object characteristics are defined in an object control unit of a data processing system. A variant model is created in the data processing system and corresponding parameters are assigned to the object and to the processing options. A visualization unit assigned to the operating device represents the complex variant model optically in a 3D-scenario tor active product configuration. The active product configuration is carried out via a display device and/or a touch screen integrated in the operating device. All processing options and degrees of freedom of the object can be animated. During the production configuration, a communication control unit effects data communication between the visualization unit and the object control unit and the configuration instance of the variant model.
[0026] In the example shown, the method for visual product configuration using interactive 3D objects is explained with reference to a car model on an operating device 1 in the form of a tablet computer comprising a touchscreen. Further see Fig 2.):
Schaarschmidt fails to explicitly teach, however, Nussbaum teaches a. selecting, via a user utilizing an internet-enabled electronic device, a type of vehicle to create a digital replica thereof (Nussbaum, abstract, the invention describes methods, systems, and computer implemented virtualization software applications and computer-implemented graphical user interface tools for remote virtual visualization of structures. Images are captured by an imaging vehicle of a structure and the captured images are transmitted to a remote server via a communication network. Using virtual 3D digital modeling software the server, using the images received from the imaging vehicle, generates a virtual 3D digital model of the structure and stores it in a database. This virtual 3D digital model can be accessed by remote users, using virtualization software applications, and used to view images of the structure. The user is able to manipulate the images and to view them from various perspectives and compare the before-the-damage images with images taken after damage have occurred. Based on all this the user is enabled to remotely communicate with an insurance agent and/or me an insurance claim.
Col 24, line 4-25, In yet further embodiments, the methods and systems described above may be applied to vehicles, such as cars, trucks, boats, motorcycles, airplanes, or trains. For example, the virtual objects may correspond to components of a vehicle, such as a windshield, a bumper, a hood, a door, a side view mirror, a wheel, a light housing, a trunk, a roof panel, or a side panel. The user may select, position, and resize the virtual objects to match damaged components of the vehicle. In some embodiments, the virtual objects may be generic virtual objects representing general components found in many vehicles. In other embodiments, the virtual objects may be selected for a specific make and model of vehicle, in order to better fit the design of the damaged vehicle. In either case, the user may select, place, position, and/or adjust the virtual objects within a virtual space representing a physical environment containing the vehicle. Pointer objects may also be added to indicate further conditions, such as broken axels or water damage. From such virtual objects or pointer objects indicated by the user, the mobile computing device may determine the extent of damage and may generate a report, as discussed above.);
Schaarschmidt and Nussbaum are analogous art, because they both teach method of creating/displaying 3D model of a vehicle/object. Nussbaum further teaches functionality of allowing user to select specific make and model of a vehicle while generate 3D model of the vehicle. Therefore, it would have been obvious to a person with ordinary skill in the art before the effective filing date of the claimed invention, to modify the 3D vehicle model (taught in Schaarschmidt), to further allow user to select make and model of the vehicle (taught in Nussbaum), so as to initiate vehicle 3D model creation from a specified generic vehicle model which best fit the targeted vehicle (Nussbaum, Col 24, line 4-25).
The combination of Schaarschmidt and Nussbaum fails to explicitly teach, however, Tech_Drone_Media teaches b. selecting, via the user utilizing the internet-enabled electronic device, one or more parameters of the digital replica (Tech_Drone_Media, the video describes method of 3D mapping of a water tower using Pix4D software/platform and DJI Mavic Pro drone.
Page 3, Pix4D is a photogrammetry software which uses images to generate point clouds, digital surface and terrain models, orthomosaics, textured models and etc.
Page 11, the software interface shows a fly route setting for the connected drone. The left side of the screen shows the altitude setup. User select 60meter as the drone’s flying altitude.
Page 12, image capture angle is set to be 10 degree.);
c. providing, via software on the internet-enabled electronic device, an indication of one or more locations to capture one or more images from via an image capture device (Tech_Drone_Media, Page 11, the software interface shows a fly route setting for the connected drone. The square box can be moved by dragging it through the image to indicate where the drone should be flying around. In the circular pattern, the drone will capture the object located at the center of the circle.
Page 12, image capture angle is set to be 10°. Therefore, an image will be capture every 10° along the circle.);
d. providing settings for the image capture device and automatically configuring the image capture device to utilize the settings (Tech_Drone_Media, page 13-15, Pix4D set up connection with the drone. The drone raised up to 60 meters and flew along the circular route around the object of interest at the pre-set interval.);
e. capturing a first image at one of the locations by the user utilizing the image capture device (Tech_Drone_Media, page 8, a plurality of captured images are listed at the bottom right of the page.);
f. capturing, another image at another of the locations by the user utilizing the image capture device (Tech_Drone_Media, page 8, a plurality of captured images are listed at the bottom right of the page.);
g. repeating steps e-g until an image for each of the locations has been captured (Tech_Drone_Media, page 8, a plurality of captured images are listed at the bottom right of the page. The square at the left side of the page indicates a plurality of waypoints at which images are captured.).
Schaarschmidt, Nussbaum and Tech_Drone_Media are analogous art, because they all teach method of creating/displaying 3D model of a vehicle/object. Tech_Drone_Media further teaches graphic user interface to connect and configure image capturing functionality for an image capture device such as a drone. Therefore, it would have been obvious to a person with ordinary skill in the art before the effective filing date of the claimed invention, to modify the 3D vehicle model (taught in Schaarschmidt and Nussbaum), to further use the 3D mapping tool/software to configure the image capturing device (taught in Tech_Drone_Media), so as to provide an intuitive platform for user to set up image capturing for an object of interest.
Regarding Claim 2. The combination of Schaarschmidt, Nussbaum and Tech_Drone_Media further teaches The method of claim 1, further comprising the step of: providing, via software on the internet-enabled electronic device, an indication of one or more levels to capture the one or more images from at the one or more locations (Tech_Drone_Media, page 13-15, Pix4D set up connection with the drone. The drone raised up to 60 meters and flew along the circular route around the object of interest at the pre-set interval.).
The reasoning for combination of Schaarschmidt, Nussbaum and Tech_Drone_Media is the same as described in Claim 1.
Regarding Claim 4. The combination of Schaarschmidt, Nussbaum and Tech_Drone_Media further teaches The method of claim 1, wherein the one or more parameters are one or more of a number of levels for the images to be captured at, a number of rows of the vehicle, a number of doors of the vehicle, whether interior images are to be captured, and whether any bookmarks should be indicated (Tech_Drone_Media, page 13-15, Pix4D set up connection with the drone. The drone raised up to 60 meters and flew along the circular route around the object of interest at the pre-set interval.).
The reasoning for combination of Schaarschmidt, Nussbaum and Tech_Drone_Media is the same as described in Claim 1.
Regarding Claim 5. The combination of Schaarschmidt, Nussbaum and Tech_Drone_Media further teaches The method of claim 1, wherein the indication is provided visually via a screen on the internet-enabled electronic device (Tech_Drone_Media, page 8, a plurality of captured images are listed at the bottom right of the page. The square at the left side of the page indicates a plurality of waypoints at which images are captured.).
The reasoning for combination of Schaarschmidt, Nussbaum and Tech_Drone_Media is the same as described in Claim 1.
Regarding Claim 7. The combination of Schaarschmidt, Nussbaum and Tech_Drone_Media further teaches The method of claim 1, wherein the image capture device is integrated with the internet enabled electronic device (Tech_Drone_Media, Page 3, Pix4D is a photogrammetry software which uses images to generate point clouds, digital surface and terrain models, orthomosaics, textured models and etc.
Page 4, Pix4D displays a list of compatible drones for connection.
Page 6, the page displays parameters of the connected Mavic Pro Drone.
Page 7, the page displays different flying pattern for the drone.
Page 11, the software interface shows a fly route setting for the connected drone. The square box can be moved by dragging it through the image to indicate where the drone should be flying around. In the circular pattern, the drone will capture the object located at the center of the circle.).
The reasoning for combination of Schaarschmidt, Nussbaum and Tech_Drone_Media is the same as described in Claim 1.
Claims 8 are rejected under 35 U.S.C. 103 as being unpatentable over
Schaarschmidt et al in view of Nussbaum et al, Tech_Drone_Media further in view of ACS (“Meet the Flying EYE 360 VR drone – The world’s first 360° VR 6K broadcast quality live streaming drone”, 2017).
Regarding Claim 8. The combination of Schaarschmidt, Nussbaum and Tech_Drone_Media fails to explicitly teach, however, ACS teaches The method of claim 1, wherein the image capture device is an external 360 degree camera (ACS, The article describes drone with 360 camera.
Page 2, par 2, The system uses the the DJI Matrice 600 Pro platform and three Blackmagic Micro Studio 4K Cameras to create images up to 6K 30fps in 10bit. The ground station provides automated stitched 4K SDI, 6K NDI, and 6KH.265 (HEVC) outputs.
Page 4, par 4, Key features from 360 Designs: 6K 360 Video
– High-quality wireless video system offers near uncompressed, low latency (<1sec), broadcast-grade 6K video. Long Range – Astonishing wireless streaming video range, over 5 miles.).
The combination of Schaarschmidt, Nussbaum and Tech_Drone_Media teaches method of creating/displaying 3D model of a vehicle/object. Tech_Drone_Media and ACS both teach 3D mapping by using drone. ACS further teaches drone with 360-degree camera. Therefore, it would have been obvious to a person with ordinary skill in the art before the effective filing date of the claimed invention, to modify the 3D vehicle model (taught in Schaarschmidt, Nussbaum and Tech_Drone_Media), to further use the drone with 360 degree camera (taught in ACS), so as to capture high-quality images/video (ACS, page 4, par 4).
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over
Schaarschmidt et al in view of Nussbaum et al, Tech_Drone_Media further in view of RCLifeOn ("3D Printing My House - Photogrammetry Using a Drone", 2017).
Claim 9 is similar in scope as Claim 1, and thus is rejected under same rationale. Claim 9 further requires:
The combination of Schaarschmidt, Nussbaum and Tech_Drone_Media further teaches h. creating an image mesh by combining each of the levels by implementing a predetermined schema (Nussbaum, Col 8, line 14-26, In performing the above-described maneuvers above and around the structure on the property 310 the imaging vehicle 320 captures multiple images, possibly hundreds or even thousands of images. From these captured images, a virtual 3D digital imaging software 260 of the remote server 210 may construct or generate a virtual 3D digital model 220 of the structure or a part thereof.
Col 21, line 24-40, At block 1906, the server 620 may communicate data including at least a portion of the virtual representation to the user electronic device 680.
Such data may be sent to the user as part or all of the virtual 3D digital model, a mesh or other encoding of a graphical representation of the model, a set of images depicting views of the model, or other representations of the physical structure based upon the virtual 3D digital model.
Col 24, line 4-25, In yet further embodiments, the methods and systems described above may be applied to vehicles, such as cars, trucks. boats. motorcycles, airplanes, or trains.); and
The reasoning for combination of Schaarschmidt, Nussbaum and Tech_Drone_Media is the same as described in Claim 1.
The combination of Schaarschmidt, Nussbaum and Tech_Drone_Media fails to explicitly teach, however, RCLifeOn teaches i. creating the digital replica of the vehicle by incorporating the plurality of captured images and the image mesh (RCLifeOn, The video describes using a drone to capture images, a software to generate the images into a 3D model and then using a 3D printer to print your house in a 3D environment.
Page 3-4, from a drone controller user interface, a point of interest is set for the drone to rotate automatically around. page 10, the captured images are all imported in Autodesk and a 3D mesh of the object is created.
Page 10, the captured images are all imported in Autodesk and a 3D mesh of the object is created.
Page 11, the virtual model of the object is created.).
Schaarschmidt, Nussbaum, Tech_Drone_Media and RCLifeOn are analogous art, because they all teach method of creating/displaying 3D model of a vehicle/object. RCLifeOn further teaches method of creating 3D mesh and model from captured plurality of images. Therefore, it would have been obvious to a person with ordinary skill in the art before the effective filing date of the claimed invention, to modify the 3D vehicle model (taught in Schaarschmidt, Nussbaum and Tech_Drone_Media), to further using the method of creating 3D mesh and model from captured plurality of images (taught in RCLifeOn), so as to provide a method for user to create 3D model for any object that can be circled around by a drone.
Claims 10-11, 13-14, 16, 18 are rejected under 35 U.S.C. 103 as being unpatentable over Schaarschmidt et al in view of Nussbaum et al, Tech_Drone_Media, RCLifeOn further in view of Holzer et al (US20200234397).
Regarding Claim 10. The combination of Schaarschmidt, Nussbaum, Tech_Drone_Media and RCLifeOn fails to explicitly teach, however, Holzer teaches The method of claim 9, further comprising the step of
J. generating a navigation grid, the navigation grid comprising a series of hyperlinks overlaid on the image mesh, wherein each hyperlink provides means for loading and rendering one of the captured images; and
k. supplementing the digital replica with the navigation grid, such that an end-user will be able to visualize and navigate through the digital replica by clicking through the navigation grid (Holzer, abstract, the invention describes a method of creating a three-dimensional (3D) skeleton based on a plurality of vertices and a plurality of faces in a two-dimensional (2D) mesh in a top-down image of an object. A correspondence mapping between a designated perspective view image and the top-down object image may be determined based on the 3D skeleton. The correspondence mapping may link a respective first location in the top-down object image to a respective second location in the designated perspective view image for each of a plurality of points in the designated perspective view image. A top-down mapped image of the object may be created by determining a first respective pixel value for each of the first locations, with each first respective pixel value being determined based on a second respective pixel value for the respective second location linked with the respective first location via the correspondence mapping.
[0041] FIG. 2 illustrates one example of a method 200 for performing geometric
analysis of a perspective view image, performed in accordance with one or more
embodiments. The method 200 may be performed on any suitable computing device.
For example, the method 200 may be performed on a mobile computing device such as
a smart phone. Alternately, or additionally, the method 200 may be performed on a
remote server in communication with a mobile computing device.
[0057] In some implementations, a mesh predicted in the camera frame may be
projected directly onto the image, obtaining a 2D mesh overlaid to the frame. Alternately, a mesh predicted in a difference frame can be used to obtain 3D-2D
correspondence mappings between mesh 3D vertices and their corresponding location
in the image, and then involve solving a Perspective-N-Point problem that yields the
transformation to the camera frame.
[0093] According to various embodiments, object component identity information
may be used to allow separate mapping estimation for different object components. For
example, the image of the vehicle shown in FIG. 12 has been segmented to identify the
different vehicle components, which are shown in different colors. For example, the vehicle includes the components 1202, 1204, and 1206 which correspond with the front
right wheel, the front right door, and the rear right door respectively. These components
have then been separately supplied as input to the machine learning algorithm for the
mapping estimation, as shown in the bottom right of the image.
Schaarschmidt, [0013] According to another feature, the present invention also
comprises a system for generating an active visual product configuration on the basis of
interactive digital 2D or 3D objects (2), which configuration can be operated on a
preferably mobile operating device having integral touchscreen (display with touch
function) and/or a pointing device, which operating device is connected via an Internet
connection to a data processing system.
[0033] If the user now activates by "touching and simultaneously dragging" the
"Rotate" button or the driver door directly as shown in FIG. 6.2, the user opens the
vehicle, and the interior containing additional vehicle objects 2 becomes visible.
Fig 7-7 .2 indicates that "seat" object is clickable for further actions. Fig 8-8.2 indicates
that "wheel" object is clickable for further actions.).
Schaarschmidt, Nussbaum, Tech_Drone_Media, RCLifeOn, and Holzer are analogous art, because they all teach method of creating/displaying 3D model of a vehicle/object. Schaarschmidt further teaches add clickable link to the model section to allow user to further interact with the 3D model. Holzer further teaches map individual point/portion of the model/mesh to an image of the corresponding portion. Therefore, it would have been obvious to a person with ordinary skill in the art before the effective filing date of the claimed invention, to modify the 3D vehicle model (taught in Schaarschmidt, Nussbaum, Tech_Drone_Media and RCLifeOn), to further map individual point/portion of the model/mesh to an image of the corresponding portion (taught in Holzer), so as to provide a more accurate/effective view for images that are captured at different viewpoints of an object (Holzer, [0005]).
Claim 11 is similar in scope as Claim 2, and thus is rejected under same rationale.
Claim 13 is similar in scope as Claim 4, and thus is rejected under same rationale.
Claim 14 is similar in scope as Claim 5, and thus is rejected under same rationale.
Claim 16 is similar in scope as Claim 7, and thus is rejected under same rationale.
Regarding Claim 18. The combination of Schaarschmidt, Nussbaum, Tech_Drone_Media, RCLifeOn, and Holzer further teaches The method of claim 10, wherein the digital replica is spinnable about a single axis by an end user manipulating the replica (RCLifeOn, page 11-15, the user use the mouse to spin the created 3D model. Further see the video at 5:57-6:08.).
Claim 17 is rejected under 35 U.S.C. 103 as being unpatentable over Schaarschmidt et al in view of Nussbaum et al, Tech_Drone_Media, RCLifeOn, Holzer et al further in view of ACS (“Meet the Flying EYE 360 VR drone – The world’s first 360° VR 6K broadcast quality live streaming drone”, 2017).
Claim 17 is similar in scope as Claim 8, and thus is rejected under same rationale.
The combination of Schaarschmidt, Nussbaum, Tech_Drone_Media, RCLifeOn and Holzer teaches method of creating/displaying 3D model of a vehicle/object. Tech_Drone_Media and ACS both teach 3D mapping by using drone. ACS further teaches drone with 360-degree camera. Therefore, it would have been obvious to a person with ordinary skill in the art before the effective filing date of the claimed invention, to modify the 3D vehicle model (taught in Schaarschmidt, Nussbaum, Tech_Drone_Media, RCLifeOn and Holzer), to further use the drone with 360 degree camera (taught in ACS), so as to capture high-quality images/video (ACS, page 4, par 4).
Allowable Subject Matter
Claims 3, 6, 12, 15 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Regarding Claim 3, it recites “The method of claim 1, step f being preceded by the step of: indicating, by the software on the internet-enabled electronic device how to reposition location and/or direction of image capture device to capture a second image” in the context of Claim 3.
The prior arts of record either alone or in combination fails to teach or suggest the above quoted limitation of Claim 3. Therefore, Claim 3 is allowable over prior art.
Regarding Claim 6, it recites “The method of claim 1, wherein the indication is provided audibly via a speaker on the internet-enabled electronic device” in the context of Claim 6.
The prior arts of record either alone or in combination fails to teach or suggest the above quoted limitation of Claim 6. Therefore, Claim 6 is allowable over prior art.
Claim 12 recites similar limitations as discussed above with regard to claim 3. Therefore, claim 12 is allowable over prior art.
Claim 15 recites similar limitations as discussed above with regard to claim 6. Therefore, claim 15 is allowable over prior art.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Pridie (US20200118329), abstract, the invention describes a system for generating a three-dimensional model of a physical object including a camera skid placed at a known distance from the physical object and moved fully around the physical object at the known distance, a set of cameras on the camera skid for capturing image data at a series of locations fully around the physical object, and a computing d vice for generating a three-dimensional model of the physical object using the known distance and the image data.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to XIN SHENG whose telephone number is (571)272-5734. The examiner can normally be reached M-F 9:30AM-3:30PM 6:00PM-8:30PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jason Chan can be reached at 5712723022. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Xin Sheng/ Primary Examiner, Art Unit 2619
Prosecution Insights