CTNF 18/970,016 CTNF 101398 DETAILED ACTION Notice of Pre-AIA or AIA Status 07-03-aia AIA 15-10-aia The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA. Claim Rejections - 35 USC § 103 07-06 AIA 15-10-15 In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status. 07-20-aia AIA The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action: A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made. 07-21-aia AIA Claim (s) 1, 3, 6, 8, 10, 13, 15, 17, and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Pavlidis et al (US 9953459 B2), Qian et al (US 20170200309 A1), and Gaal et al (US 20200263993 A1) hereinafter Pavlidis, Qian and Gaal respectively . Regarding claim 8, Pavlidis teaches a system comprising (“a system provides for a platform for storing, accessing, displaying, manipulating, updating and editing various 3D map elements.” – Col 2, Lines 1-3) : a memory; and at least one hardware processor coupled to the memory and comprising instructions that causes the system to perform operations comprising: “In certain embodiments, the modules are implemented as hardware modules/components, software modules, or any combination thereof. For example, the modules described can be software modules implemented as instructions on a non-transitory memory capable of being executed by a processor or a controller on a machine described in FIG. 5 .” – Col 5, Lines 46-51 accessing a mesh model that depicts a surface of an environment that corresponds with a location; “The map generation system 104, according to the mapping stored on the facade database, renders a 3D environment 110 for display on a viewer device 112. The 3D environment 110 is defined as a 3D map including virtual representation of physical world buildings. In another embodiment, the 3D environment 110 also includes 3D models of landscape and terrain.” – Col 5, Lines 14-20 detecting a change in the surface of the environment associated with the location; “The selected image is stored into the database in step 312 along with all of the metadata associated and learned during processing. In step 314, the system updates the 3D textures of the 3D building model with a new façade if the selected image was different than the current building image” – Col 9, Lines 48-53 NOTE: Pavlidis discloses an update process of the 3D textures of the 3D building model occurs on the condition that the selected image was different than the current building image. The 3D building model can be considered as part of the surface of the environment model associated with a location. The update condition functionally corresponds to detecting a change in the surface of an environment if the images are different. Pavlidis does not teach accessing a mesh model that depicts a surface of an environment that corresponds with a location; accessing image data from the client device; and generating an update to the mesh model based on the image data. However, Qian teaches accessing a mesh model that depicts a surface of an environment that corresponds with a location “The 3D surface model may comprise a 3D mesh model or polygonal model, where selected points in the modeling space are connected by line segments, the line segment forming polygons that represent a surface element of the overall 3D surface model.” – Par 31, Lines 6-10 NOTE: Qian further discloses that the real world locations may be used to provide locations within the 3D mesh model , see par 33, Lines 33-38. It is also further disclosed that the 3D surface mesh model location may correspond to the satellite image location of a cityscape for example, see Par 34- Lines 1-5. After the combination, the 3D mesh model depicting a surface of an environment corresponding to a location as taught by Qian can modify Pavlidis’ system for detecting a change in the surface of the 3D model of the environment associated with the location. It would have been obvious to one of ordinary skill in the art before the effective filing date of the present invention to modify Pavlidis by incorporating the teachings of Qian to access a mesh model that depicts a surface of an environment that corresponds with a location. One would be motivated to make this combination because a stored representation of the environment would provide a baseline representation of the location which will allow for detection of any changes to be incorporated into an updated model. Pavlidis in view of Qian still does not teach identifying a client device within a threshold distance of the location responsive to the change in the surface of the environment; accessing image data from the client device; and generating an update to the mesh model based on the image data. However, Gaal teaches identifying a client device within a threshold distance of the location responsive to the change in the surface of the environment; “The update model for requesting or producing sensor data comprises requesting additional data as needed to update digital map. By way of example, this need can be triggered based on detecting a change during the map data validation or based on discovering a new or previously unmapped area or road segment during map data discovery.” – Par 39, Lines 16-21 NOTE: Gaal discloses sending requests for sensor data of a specific area that was detected to have undergone a new change which functionally corresponds to identifying a client device responsive to change in the environment. This request is received by vehicles that are near the region that requires an update of the map data: “For example, data collection will be initiated as a vehicle 103 that receives the sensor data request enters the buffer region and ends after the vehicle 103 leaves the buffer region.”, see Par 38, Lines 13-16. This implies that the vehicles that capture sensor data must be within a threshold distance to receive the request to collect the sensor data for the system to update the map data. accessing image data from the client device; and generating an update to the mesh model based on the image data. “The approach further involves transmitting the sensor data request to a target number of vehicles. The target number of vehicles perform the sensor data collection event in the geographic area to collect sensor data. The approach further involves processing the sensor data to update the digital map data for the geographic area.” - Abstract NOTE: Gaal further discloses that the sensor data that is collected may refer to image date of the area, see par 35. Image data must be accessed from the client device that captured the image data in order for the system to update the mesh model. After the combination, the concept of accessing image data captured from a vehicle equipped with sensors (client device) and transmitting the collected image data back to the system to update map data as taught by Gaal can modify Pavlidis’ system for detecting changes in a model of an environment corresponding to a location. This combination would then allow Pavlidis’ system to use the image data collected by nearby client devices and update the environment model based on the changes depicted in the obtained image data. It would have been obvious to one of ordinary skill in the art before the effective filing date of the present invention to modify Pavlidis by incorporating the teachings of Gaal to identify a client device within a threshold distance of the location to changes in the surface of the environment and access image data from the client device to generate an update to the mesh model based on the image data. One would be motivated to make this combination to ensure that the system can quickly update the mesh model by using client devices that are already within the area of interest. The system would benefit from updating the mesh model as soon as a change is detected since the model of the environment should be as accurate as possible. Regarding claim 1, the claim recites similar limitations to claim 8. Therefore, method claim 1 corresponds to the system disclosed in claim 8 and is rejected for the same reasons of obviousness as used above. Regarding claim 15, the claim recites similar limitations to claim 8. Therefore, non-transitory machine-readable storage claim 15 corresponds to the system disclosed in claim 8 and is rejected for the same reasons of obviousness as used above. Regarding claim 10, Pavlidis in view of Qian and Gaal teaches the system of claim 8. Pavlidis further teaches wherein the image data is new image data, and the detecting the change in the surface of the environment includes: comparing initial image data captured at the location with the mesh model; and identifying a difference between the initial image data and the mesh model. “The selected image is stored into the database in step 312 along with all of the metadata associated and learned during processing. In step 314, the system updates the 3D textures of the 3D building model with a new façade if the selected image was different than the current building image” – Col 9, Lines 48-53 NOTE: Pavlidis discloses updating the 3D textures of a 3D building model only if the selected image data from a client device shows a difference compared to the currently stored image of the 3D building. The 3D building model can be understood as a portion that is a part of the 3D model representing a surface of the environment associated with a location. Therefore, this shows that comparing initial image data with new image data regarding changes at the location must be performed in order to update the 3D model. After the combination, the 3D mesh model representing the surface of the environment as taught by Qian can substitute the 3D building model in Pavlidis’ detection of change system for 3D environments to teach comparing the image data to the mesh model. It would have been obvious to one of ordinary skill in the art before the effective filing date of the present invention to modify Pavlidis by incorporating the teachings of Qian to compare initial image data captured at the location with the mesh model; and identifying a difference between the initial image data and the mesh model when detecting a change in the surface of the environment. One would be motivated to make this modification because a stored mesh model representation of the environment would provide a baseline representation of the location which will allow for detection of any changes to be incorporated into an updated model. Regarding claim 3, the claim recites similar limitations to claim 10. Therefore, method claim 3 corresponds to the system disclosed in claim 10 and is rejected for the same reasons of obviousness as used above. Regarding claim 17, the claim recites similar limitations to claim 10. Therefore, non-transitory machine-readable storage claim 17 corresponds to the system disclosed in claim 10 and is rejected for the same reasons of obviousness as used above. Regarding claim 13, Pavlidis in view of Qian and Gaal teach the system of claim 8. Pavlidis further teaches wherein the mesh model comprises a set of reference points, and the generating the update to the mesh model based on the image data includes: generating an updated set of reference points based on the image data. “Once a building model has been selected to correspond to the uploaded image, the uploaded image is registered to the image(s) associated with that 3D model of the building. In one embodiment, points in the uploaded image are matched accurately to points in the database. Full 3D mapping of the uploaded image as a facade of the building in the physical world is accomplished. In some embodiments, the 3D building model in the 3D map is thus re-textured and refined based on the uploaded image.” – Col 2, Lines 44-52 NOTE: Pavlidis teaches a set of points stored in the database regarding a 3D model of a building and matching the points with the set of points from the uploaded image data. The updating process which comprises of retexturing and refining the 3D building model is done using the set of points from the image data and would therefore generate an update set of points. This process can be done for the mesh model depicting the environment since a 3D building model is merely a portion of the entire environment model. After the combination, mesh model representing the surface of an environment as taught by Qian can substitute the 3D model of a building as taught by Pavlidis. This would allow Pavlidis’ system to then update the mesh model of the environment based on the received image data by generating an updated set of reference points. It would have been obvious to one of ordinary skill in the art before the effective filing date of the present invention to modify Pavlidis by incorporating the teachings of Qian to have the mesh model comprises a set of reference points and generate the update to the mesh model based on the image data by generating an updated set of reference points based on the image data. One would be motivated to make this modification because the stored reference points gives an indication to the structure of the mesh model and by comparing it with the image data, the generated set of updated points will more accurately reflect the changes depicted in the image data. Regarding claim 6, the claim recites similar limitations to claim 13. Therefore, method claim 2 corresponds to the system disclosed in claim 13 and is rejected for the same reasons of obviousness as used above. Regarding claim 20, the claim recites similar limitations to claim 13. Therefore, non-transitory machine-readable storage claim 20 corresponds to the system disclosed in claim 13 and is rejected for the same reasons of obviousness as used above . 07-21-aia AIA Claim (s) 2, 9, 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Pavlidis, Qian and Gaal, and Lookingbill et al (US 8818081 B1) hereinafter Lookingbill . Regarding claim 9, Pavlidis in view of Qian and Gaal teaches the system of claim 8. Pavlidis does not teach wherein the generating the update to the mesh model based on the image data includes: accessing a three-dimensional (3D) model that corresponds with an object associated with the location. However, Lookingbill teaches wherein the generating the update to the mesh model based on the image data includes: accessing a three-dimensional (3D) model that corresponds with an object associated with the location, “The computing devices may receive the captured video data including information representing visual orientation and positioning information associated with the captured video data, and a stored data model representing a 3D geometry depicting objects associated with the location may be accessed” – Col 1, Lines 27-31 NOTE: After the combination, the 3D model corresponding with an object associated to the location as taught by Lookingbill can be part of the 3D mesh model corresponding to an environment taught by Pavlidis in view of Qian. the 3D model comprising the mesh model that depicts the surface of the location, By receiving video segments from a large number of users at a given location, an accurate and up-to-date 3D model of the location can be maintained. In this regard, a visual correlation can be determined between one or more objects depicted in the video segments and projections (e.g., reconstructed target images) of the 3D model. These projections of the 3D model depict objects that are associated with the location.” – Col 3, Lines 35-41 NOTE: Lookingbill discloses using captured crowdsourced images from video data and a stored model of a 3D geometry depicting objects associated with the location to update a 3D model of a location. The images are used to update or add to the 3D model. This functionally corresponds to the 3D model depicting an object corresponding to the mesh model depicting the surface of the location. After the combination, the stored 3D data model representing a 3D object as taught by Lookingbill can correspond with an object associated with the 3D model of an environment as taught by Pavlidis. wherein a portion of the mesh model corresponds with a position of the object within the location; In one example, the positioning information associated with the captured video data overlaps with positioning information for the 3D geometry. In this example, a visual correlation may be determined between one or more objects depicted in the captured video data and one or more objects depicted in the projections.” – Col 1, Lines 43-48 NOTE: Lookingbill discloses position information of the 3D geometry which would naturally correspond to the position of the object within the location since the 3D object would be within the 3D model of the location. generating the update to the portion of the mesh model that corresponds with the position of the object within the location based on image attributes of the image data. “FIG. 7 illustrates an updating technique 700 that may be used, for example, to update a data model of a location. To update a data model of a location stored in a data model database 314, one more input images 508 may be integrated into a 3D geometry of images 416 associated with the data model. As discussed above, this update may be based on the visual orientation and positioning information associated with the input images 508 and similar information for corresponding images from the 3D geometry 416” – Col 7, Lines 1-9 NOTE: Lookingbill discloses using position information associated with input images corresponding to the 3D object associated with the location for updating the 3D model of the location. After the combination, the use of position information of the object within the location for updating the 3D mesh model of the location as taught by Lookingbill can modify Pavlidis’ updating process so that the Pavlidis’ system can retexture and refine the 3D building models in the portion of the 3D environment corresponding to the position of the object. It would have been obvious to one of ordinary skill before the effective filing date of the present invention to modify Pavlidis by incorporating the teachings of Lookingbill to update the 3D mesh model of the accessing the 3D model of the object associated with the model depicting the surface of the location by using the position of the object within the location to update a portion of the mesh model. One would be motivated to make this combination so that the update process only occurs in a region that is necessary which saves computation resources compared to updating the mesh model in its entirety. Regarding claim 2, the claim recites similar limitations to claim 9. Therefore, method claim 2 corresponds to the system disclosed in claim 9 and is rejected for the same reasons of obviousness as used above. Regarding claim 16, the claim recites similar limitations to claim 9. Therefore, non-transitory machine-readable storage claim 16 corresponds to the system disclosed in claim 9 and is rejected for the same reasons of obviousness as used above . 07-21-aia AIA Claim (s) 4, 11, 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Pavlidis, Qian, Gaal, and Fleischman et al (US 20200154095 A1), hereinafter Fleischman . Regarding claim 11, Pavlidis in view of Qian and Gaal teaches the system of claim 8. Pavlidis does not teach wherein the image data comprises a timestamp, and wherein the accessing the image data is based on the timestamp. However, Fleischman teaches wherein the image data comprises a timestamp, and wherein the accessing the image data is based on the timestamp. “During the process of recording video by the video capture system, the client device 150 captures an image and records its timestamp 162.” – Col 5, Lines 1-3 NOTE: After the combination, the client devices capturing an image and recording a timestamp as taught by Fleischman can be added to Pavlidis’ device so that the image data received by the client device sources may also include timestamp data. This would then allow Pavlidis’ system to access time information when analyzing image date from client devices. It would have been obvious to one of ordinary skill in the art before the effective filing date of the present invention to modify Pavlidis by incorporating the teachings of Fleischman to have the image data comprise a timestamp. One would be motivated to make this combination to compare the currently stored surface of the environment at a certain time with an image data received from a client device captured at a later time. This will ensure that the update of the 3D mesh model using the image data received from the client device will reflect the changes that occurred. Regarding claim 4, the claim recites similar limitations to claim 11. Therefore, method claim 4 corresponds to the system disclosed in claim 11 and is rejected for the same reasons of obviousness as used above. Regarding claim 18, the claim recites similar limitations to claim 11. Therefore, non-transitory machine-readable storage claim 18 corresponds to the system disclosed in claim 11 and is rejected for the same reasons of obviousness as used above . 07-21-aia AIA Claim (s) 5, 12, 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Pavlidis, Qian Gaal, and Salowitz (US 20180299272 A1), hereinafter Salowitz . Regarding claim 12, Pavlidis in view of Qian and Gaal teaches the system of claim 8. Pavlidis does not teach wherein the identifying the client device within the threshold distance includes: detecting the client device within the threshold distance of the location; and causing display of a notification at the client device to guide a user of the client device to the location. However, Gaal teaches wherein the identifying the client device within the threshold distance includes: detecting the client device within the threshold distance of the location; and causing display of a notification at the client device to guide a user of the client device to the location. For example, data collection will be initiated as a vehicle 103 that receives the sensor data request enters the buffer region and ends after the vehicle 103 leaves the buffer region.” - Par 38, Lines 13-16 NOTE: Gaal discloses that only vehicles within the range of the specified area are transmitted the request to capture image data. This functionally corresponds to identifying a client device within the threshold distance of the location It would have been obvious to one of ordinary skill in the art before the effective filing date of the present invention to modify Pavlidis by incorporating the teachings of Gaal to detect client devices within the threshold distance of the location responsive to the change in the surface of the environment. One would be motivated to make this combination to find nearby devices that may assist in obtaining image data that reflects the changes to the surface of the environment. Pavlidis still does not teach wherein the identifying the client device within the threshold distance includes: detecting the client device within the threshold distance of the location; and causing display of a notification at the client device to guide a user of the client device to the location. However, Salowitz teaches wherein the identifying the client device within the threshold distance includes: detecting the client device within the threshold distance of the location; and causing display of a notification at the client device to guide a user of the client device to the location. “determine that the user is within a threshold proximity to a nearby familiar location on the list of familiar locations, and in response, output directional information indicating a relative position of the nearby familiar location on the list of familiar locations.” - Abstract NOTE: Salowitz discloses outputting directional information to guide the user to a familiar location and that to guide the user to the location the device may output textual and/or audio notifications: “rather than or in addition to displaying graphical indicators, a device may output textual and/or audio notifications.”, see Par 32, Lines 14-19. It is not explicitly taught that the user is being guided to the location responsive to the change in the environment. However after the combination, the method of guiding the user to a location by displaying of a notification in response to a proximity threshold distance as taught by Salowitz can modify Gaal’s method for identifying a client device within a threshold distance of the location responsive to the change in the surface of the environment. This modification can be further added to Pavlidis’ system for detecting a change in the surface of the environment. associated with the location. This will then allow Pavlidis’ system to identify a client device within a threshold distance of the location responsive to change and then display a notification to the client device to guide the user to the location. It would have been obvious to one of ordinary skill in the art before the effective filing date of the present invention to modify Pavlidis by incorporating the teachings of Salowitz to cause displaying of a notification at the client device to guide a user of the client device to the location. One would be motivated to make this combination to guide the user to where the change in the surface of the environment corresponding with a location is present so that image data can be captured and sent to the system to update the 3D mesh model of the location. Regarding claim 5, the claim recites similar limitations to claim 12. Therefore, method claim 5 corresponds to the system disclosed in claim 19 and is rejected for the same reasons of obviousness as used above. Regarding claim 19, the claim recites similar limitations to claim 12. Therefore, non-transitory machine-readable storage claim 19 corresponds to the system disclosed in claim 12 and is rejected for the same reasons of obviousness as used above . 07-21-aia AIA Claim (s) 7 and 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Pavlidis, Qian, Gaal, and Bare et al (US 20190096138 A1), hereinafter Bare . Regarding claim 14, Pavlidis in view of Qian and Gaal teach the system of claim 8. Pavlidis further teaches receiving the image data from the client device. “Image acquisition module 201 accepts images and image metadata from many sources, including but not limited to: orthographic and oblique aerial and satellite imagery, terrestrial vehicular-collected imagery and terrestrial mobile user imagery (e.g., crowd sourced) from smartphone cameras, wearable cameras, other digital cameras, web-cams, security footage and other camera systems.” – Col 6, Lines 16-22 NOTE: Pavlidis discloses receiving images from many client device sources. Examples being images captured from smartphones or wearable cameras. Pavlidis does not teach causing display of Augmented-Reality (AR) content that identifies the change in the surface of the environment at the client device; and receiving the image data from the client device. However, Bare teaches causing display of Augmented-Reality (AR) content that identifies the change in the surface of the environment at the client device “The method includes receiving from the mobile interface device a request for target object information associated with a target object in the dynamic structural environment. A pose of the mobile interface device relative to the target object is determined accounting for spatial differences in the environment coordinate system resulting from changes in the dynamic structure. The method also includes assembling AR target object information for transmission to and display on the mobile interface device and transmitting the AR target object information to the mobile interface device” – Abstract NOTE: Bare discloses a method of transmitting augmented reality content to a mobile interface device to display the changes of the object’s position or condition in an environment. AR information is viewed by the client device in the form of an overlay over the image of an object, see par 57. The object in an environment can be understood as portion of the surface of the environment. This functionally corresponds to using AR content to display to the client device changes in the surface of the environment. After the combination the method of using AR content to display changes in the surface of an environment to a client device can be added to Pavlidis’ system for detecting changes in the surface of the environment associated with a location. This would then allow Pavlidis’ system to display AR content depicting the changes that were identified on the surface of the environment of the associated location. It would have been obvious to one of ordinary skill in the art before the effective filing date of the present invention to modify Pavlidis by incorporating the teachings of Bare to display of AR content that identifies the change in the surface of the environment at the client device. One would be motivated to make this combination to provide an intuitive w for the user of a client device to clearly identify the changes in the surface of the environment associated with a location. This would aid in collecting image data that will accurately update the 3D mesh model of the location. Regarding claim 7, the claim recites similar limitations to claim 14. Therefore, method claim 7 corresponds to the system disclosed in claim 14 and is rejected for the same reasons of obviousness as used above. Conclusion Any inquiry concerning this communication or earlier communications from the examiner should be directed to DAVID V. NGUYEN whose telephone number is (571)272-6111. The examiner can normally be reached M-F 9:00-5:00. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, King Y Poon can be reached at 571-270-0728. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. /DAVID VAN NGUYEN/ Examiner, Art Unit 2617 /KING Y POON/Supervisory Patent Examiner, Art Unit 2617 Application/Control Number: 18/970,016 Page 2 Art Unit: 2617 Application/Control Number: 18/970,016 Page 3 Art Unit: 2617 Application/Control Number: 18/970,016 Page 4 Art Unit: 2617 Application/Control Number: 18/970,016 Page 5 Art Unit: 2617 Application/Control Number: 18/970,016 Page 6 Art Unit: 2617 Application/Control Number: 18/970,016 Page 7 Art Unit: 2617 Application/Control Number: 18/970,016 Page 8 Art Unit: 2617 Application/Control Number: 18/970,016 Page 9 Art Unit: 2617 Application/Control Number: 18/970,016 Page 11 Art Unit: 2617 Application/Control Number: 18/970,016 Page 12 Art Unit: 2617 Application/Control Number: 18/970,016 Page 13 Art Unit: 2617 Application/Control Number: 18/970,016 Page 15 Art Unit: 2617 Application/Control Number: 18/970,016 Page 16 Art Unit: 2617 Application/Control Number: 18/970,016 Page 17 Art Unit: 2617 Application/Control Number: 18/970,016 Page 18 Art Unit: 2617 Application/Control Number: 18/970,016 Page 19 Art Unit: 2617
Prosecution Insights