DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments and amendments in the Amendment filed August 6, 2026 (herein “Amendment”), with respect to the rejection of claims 14–20 under 35 U.S.C. 101 as being directed towards non-statutory subject matter have been fully considered and are persuasive. Specifically, Applicant has amended the Specification to now clearly define term computer readable storage media to not be transitory signals per se. Accordingly, the rejection of claims 14–20 under 35 U.S.C. 101 has been withdrawn.
Applicant’s arguments and amendments in the Amendment, with respect to the rejections of claims 1, 8 and 14, and various claims depending therefrom under 35 U.S.C. 103 have been fully considered and are persuasive in part. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground of rejection is made in view of Foco et al., US Patent Application Publication No. US 2024/0203052 A1.
Specifically, Applicant’s arguments on pages 13–14 (a section A of the arguments), and the arguments under section C on pages 15–16 of the Amendment are persuasive regarding the combination of Herman and Robinson not teaching or suggesting the newly amended limitations.
However, Applicant’s arguments on pages 14–15, Section B, and on pages 16–17, Section D are not persuasive. Responding to the Section D arguments first, which are that neither Herman nor Robinson teaches the claimed “removing all details prevents leakage of confidential information,” first it is noted that contrary to applicant’s statement that the wherein clause is not “merely” a statement of intended purpose, the limitation does not limit the claim by providing the method steps that would realize a result of “prevents leakage of confidential information.” Second, even if the result were to be limiting, simply “preventing leaking of confidential information” by way of removing all details except relevant points, simply requires prevention not that actual leakage never happens. Accordingly, the previously set forth combination of Herman and Robinson—especially Herman’s deleting of a portion of data points corresponding to an object with personally identifiable information (PII)—at least “prevents” leaking of confidential information by way of removing all sensitive details in the data points. Therefore, Applicant’s remarks under section D are not persuasive.
Responding to Applicant’s remarks under section B, that Robinson is not properly combinable with Hernan as the stated motivation for doing so set forth in the Non-Final Action issued July 23, 2026 (herein “Non-Final Action”) on page 6, as “doing so would allow for a connection between consumers of image data specific to certain objects, thus providing exchange with other users in mutual/shared activities,” is contended by applicant to be “entirely foreign to the subject matter of the claims.” First, it is noted that the motivation to combine prior art references is with respect to the teachings of the prior art references, and not with respect to “the subject matter of the claims,” as applicant contends. Besides, assuming arguendo that Applicant’s contention that Robinson’s field of application is directed towards consumer sharing, social exchange or advertising content delivery, this field of application is not mutually exclusive of privacy-preserving image abstraction in a smart vehicle context. Even smart vehicle data in 2024 would be understood by a PHOSITA as having value in consumer sharing for advertising content.
Further, Applicant sets forth on the middle of page 15 of the Amendment under section B that “importing Robinson’s content insertion technique into Herman’s point cloud density reduction system would require a fundamental restructuring of Herman’s operating principle.” Under this rationale, first applicant must objectively determine what the principle of operation is for the reference being modified, then properly restate the modification of record as having changed the principle of operation. See MPEP §2143.01(V). However, the Applicant has not set forth either Herman’s principle of operation (which, taking from Herman’s title, would be “Anonymizing personally identifiable information in sensor data”) nor has Applicant properly restated the modification of record, which was set forth on page 6 of the Non-Final Action to be that the insertion of object data into an environmental image of Robinson is combined with Herman’s point cloud visual data density reduction for keeping secure PII. Accordingly, such a combination of merely inserting object data into an environmental image would not change Herman’s objective of anonymizing PII in sensor data.
Therefore, the portions of the rejection relying on Herman and Robinson are maintained in this action as Applicant’s arguments, while fully considered, are not persuasive. For the newly recited limitations, newly cited Foco is relied upon in a new grounds of rejection.
Applicant's arguments and amendments in the Amendment, with respect to the rejection of claims 3–5, 10–11 and 16–18 under 35 U.S.C. 103 have been fully considered but they are not persuasive. Specifically, Applicant argues that third reference Ravella does not teach or suggest all of the limitations of “receiving information regarding analysis of the abstracted image of each of the one or more of the set of objects in the environment surrounding a smart vehicle with the contextual information regarding the environment attached as the metadata from the set of data analysis services; and operating functional components of the smart vehicle automatically based on the information regarding the analysis of the abstracted image of each of the one or more of the set of objects in the environment surrounding the smart vehicle,” which is most of claim 3. However, as set forth on the bottom of page 9 to the top of page 10 of the Non-Final Action issued July 23, 2026, Ravella is relied upon for only the limitations regarding explicit teachings of “receiving information regarding analysis of the image” where the information is “from the set of data analysis services,” since as explained the deficiency of primary reference Herman was simply that Herman suggested that the information regarding the analysis of the image is received somewhere in that Herman does teach using abstracted data for further analysis. Herman just doesn’t explicitly teach receiving information regarding analysis of the image from the set of data analysis services, and for this limited reliance, Ravella was cited. Accordingly, Applicant’s arguments that Ravella doesn’t teach “analysis of an abstracted image with contextual metadata attached in the specific manner the claims require” is not only non-responsive to the combination set forth in the rejection, which includes limited reliance upon Ravella, but also mischaracterizes what is actually claimed. That is, claim 3, in the relevant portion at issue requires simply that “information regarding analysis of the abstracted image” is received, where Ravella is relied upon just to provide teachings of “receiving information” regarding analysis of an image, met by Ravella’s teachings of a server that collects and analyzes images from a vehicle camera and provides driving solutions (the result of analysis of the images) to individual vehicles—that is, the individual vehicles are receiving information regarding analysis of an image from the set of data analysis services on the server.
Applicant further sets forth on page 18 of the Amendment that “The structural prerequisite of a privacy-abstracted image with attached environmental metadata, as recited in claim 2 from which claim 3 depends, is absent from Ravella, and cannot be supplied by that reference standing alone.” Indeed, the test for obviousness is not whether the features of a secondary reference may be bodily incorporated into the structure of the primary reference; nor is it that the claimed invention must be expressly suggested in any one or all of the references (i.e. to Applicant’s point about “cannot be supplied by [a] reference standing alone”). Rather, the test is what the combined teachings of the references would have suggested to those of ordinary skill in the art. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981). Accordingly, the rejection of claim 3 is set forth in view of the combination of the combined teachings of Herman, Robinson and Ravella, and as such since Applicant’s remarks focused only on Ravella, they are not persuasive, and the rejection of claim 3, and claims 10 and 16 reciting similar limitations, and any claims depending therefrom is herein maintained.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1–2, 6–9, 12–15 and 19–20 are rejected under 35 U.S.C. 103 as being unpatentable over Herman et al., US Patent Application Publication No. US 2023/0375707 A1 (herein “Herman”) in view of Robinson et al., US Patent No. 8,316,450 B2 (herein “Robinson”) in view of Foco et al., US Patent Application Publication No. US 2024/0203052 A1.
Regarding claims 1, 8 and 14, with substantive differences between the claims noted in curly brackets {}, deficiencies of Herman noted in square brackets, and with claim 1 as exemplary, Herman teaches {a method comprising: - claim 1 / A smart vehicle system comprising: a processor set; one or more computer-readable storage media; and program instructions stored on the one or more computer-readable storage media to cause the processor set to perform operations comprising: - claim 8 / A computer program product comprising: one or more computer-readable storage media; and program instructions stored on the one or more computer-readable storage media to perform operations comprising: - claim 14} (Herman Abstract, a computer including a processor and a memory storing instructions to execute by the processor, lidar data processing steps)
identifying a plurality of [predefined] relevant points [for a type of object] on each of one or more of a set of objects captured in [an image] of an environment needing to be abstracted (Herman ¶¶ 37–38, 40–43 and 48, lidar data captured from a vehicle computer and from which a point cloud is generated, the point cloud including sensitive personally identifiable information PII (needing to be abstracted) such as a license plate (one or more of a set of objects) comprised of specific points in the point cloud, where the points belonging to the PII objects have a point density that is reduced, and where the reduced density point cloud is transmitted to a remote computer for further processing, thus remaining points from the reduced density of the PII object being identified as relevant);
[extracting the plurality of predefined relevant points from each of the one or more of the set of objects captured in the image of the environment needing to be abstracted;]
[inserting the extracted plurality of relevant points] corresponding to each of the one or more of the set of objects into [the image] of the environment forming an abstraction of each of the one or more of the set of objects needing to be abstracted (Herman ¶¶40 and 43, fig. 2, while reducing the density of the point cloud by deleting some points in a region 210 corresponding to an object 205 with PII, thus leaving only certain points (relevant points), the density and points of the point cloud areas other than region 210 such as region 215 that do not have PII, including points surrounding and outside of region 210 (the environment) retained/unchanged); and
removing all details in [the image] of the environment except for the plurality of relevant points corresponding to each of the one or more of the set of objects inserted into [the image], wherein removing all details prevents leakage of confidential information from the [image] of the environment, to form an abstracted [image] of each of the one or more of the set of objects in the environment to preserve confidentiality (Herman ¶¶ 44 and 41, a geometric blurring or random noise (both which remove details as it blurs and makes details imperceptible) is applied to a subset of the points in the region 210, leaving alone (except for) the remaining points that meet the desired threshold density for ensuring that the resulting density is not enough to identify a person or perform text recognition (preserve confidentiality and prevent leakage of confidential information)).
While Herman discloses its point cloud abstraction process to retain surrounding environment points in a point cloud and simply to delete some of the points in the PII object “needing to be abstracted,” in such a process, while the end result of having a point cloud where only relevant points for objects needing abstraction are present, nonetheless, Herman does not explicitly teach “inserting” the relevant points into “the image” of the environment. Further, while Herman discloses lidar based point clouds, Herman does not explicitly teach images, although.
However, Robinson teaches inserting graphical data (which would include pixels/points) of an object into an image of an environment (Robinson col. 4, ll. 60-65, col. 21, l. 60 – col. 22, l. 11, fig. 14e, objects as images inserted at a marker in the content, where the content is an image of an environment).
Further, as noted above in square brackets, Herman does not explicitly teach, where Foco teaches predefined relevant points for a type of object (Foco ¶¶37–38, a sparse model of keypoints comprised of subsets of points in an environment belonging to individual objects is determined and registered to be retained and used for tracking and registration over time (thus predefined for future registration use)), extracting the plurality of predefined relevant points from each of the one or more of the set of objects captured in the image of the environment needing to be abstracted (Foco ¶¶30–31, objects shape data for an identified object such as a chair, as a mesh, model or other representation are placed in the environment in place of (extracting) the generated segmentation, where for example, generated segmentation of chairs in a image (environment) are replaced with the representations 222 from an object database which are accurate representations of those objects that have been generated and verified for the physical items (predefined relevant points), and where ¶32 teaches that such objects can be pedestrians in an urban roadway environment, where a pedestrian would have a privacy need to be abstracted), and inserting the extracted plurality of relevant points (Foco ¶¶30–31, the environment enables objects predefined as mesh, point cloud, or model, to be added (inserted) or otherwise modified to create an updated view of the environment representation).
Therefore, taking the teachings of Herman and Robinson as a whole, it would have been obvious to a person having ordinary skill in the art (herein “PHOSITA”) before the effective filing date of the claimed invention to have modified the point cloud visual data density reduction for keeping secure PII method of Herman with the insertion of objects data into an environmental image disclosed in Robinson at least because doing so would allow for a connection between consumers of image data specific to certain objects, thus providing exchange with other users in mutual/shared activities. See Robinson col. 3, ll. 29–35, and col. 22, ll. 12–21.
Therefore, taking the teachings of Herman as modified above and Foco together as a whole, it would have been obvious to a PHOSITA before the effective filing date of the claimed invention to have modified the point cloud visual data density reduction for keeping secure PII method of Herman with the extraction of relevant points from objects captured in an image and the insertion of those extracted points into an environmental as disclosed in Foco at least because doing so would allow for a vehicle or other autonomous machinery having a safety system to make better decisions by understanding the types of objects nearby and which ones are likely to affect navigation and have to be monitored. See Foco ¶32 .
Regarding claims 2, 9 and 15, with claim 2 as exemplary, and deficiencies of Herman noted in square brackets, Herman teaches further comprising: [attaching contextual information regarding the environment as metadata to the] abstracted [image of each of the one or more of the set of objects in the environment]; and sending the abstracted [image] of each of the one or more of the set of objects in the environment with the contextual information regarding the environment [attached as the metadata] to a set of data analysis services (Herman ¶¶ 5, 28, 48 and 51, a transceiver on the vehicle provides communication from the vehicle to a remote computer 110, the communication including the lidar data with the specific to PII object region reduced (abstracted) point cloud data to the remote computer 110 for further analysis such as performance of an advanced driver assistance systems (ADAS) feature of the vehicle).
Herman does not specifically teach where Robinson teaches attaching contextual information regarding the environment as metadata to the image of each of the one or more of the set of objects in the environment; and contextual information regarding the environment attached as the metadata (Robinson col. 3, ll. 62–65, col. 4, ll. 30–44, col. 17, ll.9–50, claim 1, metadata describing markers, objects and content are part of the data and processed for the markers, objects and content, an thus when an object is downloaded and inserted into content, the metadata is attached thereto, the metadata including data corresponding to one or both of a temporal and spatial positioning of the marker within a viewer perceivable portion of the content (thus contextual regarding the environment)).
Therefore, taking the teachings of Herman and Robinson as a whole, it would have been obvious to a PHOSITA before the effective filing date of the claimed invention to have modified the point cloud visual data density reduction for keeping secure PII method of Herman with the insertion of objects data including metadata into an environmental image as disclosed by Robinson at least because doing so would allow for a connection between consumers of image data specific to certain objects, thus providing exchange with other users in mutual/shared activities. See Robinson col. 3, ll. 29–35, and col. 22, ll. 12–21.
Regarding claims 6, 12 and 19, with claim 6 as exemplary, Herman teaches further comprising: determining whether the image of the environment captures the set of objects based on performing an analysis of the image (Herman ¶¶37–40, object recognition on the point cloud data fused with camera data (images) is performed and a CNN outputs the detected object type and whether that object type is associated with PII); and responsive to determining that the image of the environment does capture the set of objects based on the analysis of the image, applying a set of confidentiality criteria to the set of objects captured in the image of the environment (Herman ¶¶40–41, objects with PII are then evaluated to see if the point cloud density is above a certain threshold associated with being able to discern the PII (confidentiality criteria)).
Regarding claims 7, 13 and 20, with claim 7 as exemplary, Herman teaches further comprising: determining whether the one or more of the set of objects captured in the image of the environment need to be abstracted based on applying a set of confidentiality criteria (Herman ¶¶40–43, objects with PII are then evaluated to see if the point cloud density is above a certain threshold associated with being able to discern the PII (confidentiality criteria), and of they are, then the point cloud is processed to reduce the density in the region where the object having PII exists); and responsive to determining that the one or more of the set of objects captured in the image of the environment do need to be abstracted based on applying the set of confidentiality criteria, identifying the plurality of relevant points on each of the one or more of the set of objects captured in the image of the environment needing to be abstracted (Herman ¶¶42–43, the density of the point cloud in the region with the object having PII is reduced, therefore the remaining points in the point cloud region being identified as relevant points).
Claims 3–5, 10–11 and 16–18 are rejected under 35 U.S.C. 103 as being unpatentable over Herman in view of Robinson and Foco, as set forth above regarding the independent claims, and further in view of Ravella et al., US Patent Application Publication No. 2025/0124794 A1 (herein “Ravella”).
Regarding claims 3, 10 and 16, with deficiencies of Herman noted in square brackets [] and with claim 3 as exemplary, Herman teaches further comprising: [receiving information regarding] analysis of the abstracted [image] of each of the one or more of the set of objects in the environment surrounding a smart vehicle [with the contextual information regarding the environment attached as the metadata from the set of data analysis services] (Herman ¶¶51–56, lidar data relating to the point cloud of objects with PII that has been reduced in point density for those objects is saved and used for further analysis including automatic driving assistance for the vehicle); and
operating functional components of the smart vehicle automatically based on the information regarding the analysis of the abstracted image of each of the one or more of the set of objects in the environment surrounding the smart vehicle (Herman ¶51, further analysis for performance of an ADAS (operating functional components of the smart vehicle automatically) feature of the vehicle and for a particular object in the lidar data) [with the contextual information regarding the environment attached as the metadata received from the set of data analysis services].
As noted above, Herman does not, however Robinson at least teaches with the contextual information regarding the environment attached as the metadata (Robinson col. 3, ll. 62–65, col. 4, ll. 30–44, col. 17, ll.9–50, claim 1, metadata describing markers, objects and content are part of the data and processed for the markers, objects and content, an thus when an object is downloaded and inserted into content, the metadata is attached thereto, the metadata including data corresponding to one or both of a temporal and spatial positioning of the marker within a viewer perceivable portion of the content (thus contextual regarding the environment)).
Further, although Herman teaches for example in ¶51 that the abstracted data can be used for further analysis and stored by the remote computer for automated driving assistance in the vehicle, nonetheless Herman does not explicitly teach that the remote compute sends analysis information back to the vehicle, and therefore Herman does not explicitly teach where Ravella teaches receiving information regarding analysis of the image from the set of data analysis services (Ravella fig. 1, ¶¶58–59, 69, 139, 197, 205–207, server 104 collects and analyzes vehicle data, including data from cameras (images), and provides driving solutions to individual vehicles 102 including vehicle control, as well as performing the operations of computer system 600, including providing to vehicles via a wireless network, trajectory data for the vehicle from analyzing the sensor data).
Therefore, taking the teachings of Herman and Robinson as a whole, it would have been obvious to a PHOSITA before the effective filing date of the claimed invention to have modified the point cloud visual data density reduction for keeping secure PII method of Herman with the insertion of objects data including metadata into an environmental image as disclosed by Robinson at least because doing so would allow for a connection between consumers of image data specific to certain objects, thus providing exchange with other users in mutual/shared activities. See Robinson col. 3, ll. 29–35, and col. 22, ll. 12–21.
Further, taking the teachings of Herman as modified by Robinson and Ravella altogether as a whole, it would have been obvious to a PHOSITA before the effective filing date of the claimed invention to have modified the advanced drive assistance system of Herman with the analysis and trajectory information transmitted back to a vehicle from the server as disclosed in Ravella at least because doing so would improve decision making in autonomous vehicles. See Ravella ¶5.
Also further regarding claim 10, and regarding claims 4 and 17, Herman does not explicitly teach, where Ravella teaches wherein the information includes a prediction regarding movement of each of the one or more of the set of objects in the environment surrounding the smart vehicle in relation to speed and direction of movement of the smart vehicle (Ravella ¶¶ 205–213, the travel trajectory to the vehicle sent by the computer system/server includes scene perception and prediction regarding other vehicles in a zone of interest, including other vehicles coming at high speed from the back of the vehicle (spend and direction of movement relative to the smart vehicle)).
Therefore, taking the teachings of Herman as modified by Robinson and Ravella altogether as a whole, it would have been obvious to a PHOSITA before the effective filing date of the claimed invention to have modified the advanced drive assistance system of Herman with the analysis and trajectory information transmitted back to a vehicle from the server as disclosed in Ravella at least because doing so would improve decision making in autonomous vehicles. See Ravella ¶5.
Regarding claims 5, 11 and 18, Herman does not explicitly teach but Ravella teaches further comprising: capturing the image of the environment surrounding the smart vehicle along with contextual information regarding the environment using an Internet of Things sensor set (Ravella ¶¶54–58, each vehicle having sensors including global positioning sensors, lidar scanner, radar sensor, IMU sensors reporting acceleration, orientation, angular rates, odometry sensor, and one or more cameras (capturing an image) of the environment around the vehicle, where the vehicle data is provided to servers, where ¶71 teaches the vehicle (and thus its sensors) communicating with the server via the Internet, thus the sensors being “Internet of Things” sensors, also ¶145 installation sensors providing data to the vehicle on information pertaining to an intersection the vehicle is present within), the contextual information includes time of day (Ravella ¶147, intersection information including the time at which the vehicle entered the current mode), geographic location (Ravella ¶54, GPS sensors provide a geographical position of the vehicle), vehicle speed, and vehicle direction of movement of the smart vehicle (Ravella ¶69, accelerometers and gyroscopes to measure the position, the orientation, and the speed of the vehicle, and also see ¶¶147-148, intersection information including a lane travel direction for the lane the vehicle is in); and performing an analysis of the image of the environment surrounding the smart vehicle using computer vision and a set of machine learning models (Ravella ¶61, image data collected by the cameras is processed using an object detection model, and ¶71 machine learning models processing the vehicle data).
Therefore, taking the teachings of Herman as modified by Robinson and Ravella altogether as a whole, it would have been obvious to a PHOSITA before the effective filing date of the claimed invention to have modified the advanced drive assistance system of Herman with the sensors and data and processing models used as disclosed in Ravella at least because doing so would improve decision making in autonomous vehicles. See Ravella ¶5.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MICHELLE M KOETH whose telephone number is (571)272-5908. The examiner can normally be reached Monday-Thursday, 09:00-17:00, Friday 09:00-13:00, EDT/EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Vincent Rudolph can be reached at 571-272-8243. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
MICHELLE M. KOETH
Primary Examiner
Art Unit 2671
/MICHELLE M KOETH/Primary Examiner, Art Unit 2671
Prosecution Insights