DETAILED ACTION
AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
Claims 1, 9 and 12-14 are amended, claim 2 is canceled and claim 15 is added.
Accordingly, claims 1 and 3-15 are pending.
Claims 1 and 3-15 are rejected (Final Rejection).
Response to Amendments and Arguments
Applicant’s arguments, at Pages 9-13, filed 04/24/2026, with respect to the rejections under 35 U.S.C. § 102 and 103 have been fully considered but are moot because the new ground of rejection, necessitated by Applicant’s amendment, does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
In response to applicant's argument that the references fail to show certain features of the invention, it is noted that the features upon which applicant relies (i.e., “providing different metadata to the first data sample set and the second data sample set”) are not recited in the rejected claim(s). Although the claims are interpreted in light of the specification, limitations from the specification are not read into the claims. See In re Van Geuns, 988 F.2d 1181, 26 USPQ2d 1057 (Fed. Cir. 1993).
Claim Rejections - 35 U.S.C. § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1 and 3-14 are rejected under 35 U.S.C. § 103 as being unpatentable over ZHANG et al. (CN 113743603), hereinafter “ZHANG”, which refers to the English language translation of ZHANG et al., in view of SHEU et al. (U.S. Patent Application Publication No. 2022/0027684 A1).
Examiner’s Note: Although the claims refer to “electronic devices” broadly, the specification mentions sensor approximately 200 times and the drawings use “sensor” data as an example of the “electronic device” data. Thus, sensor data is interpreted as corresponding to electronic device data/output.
Regarding claim 1, ZHANG discloses an information processing device comprising: one or more processors (processor, Para. 9 of Page 5 of ZHANG) configured to: perform, by using a model for performing a simulation of operations of a plurality of electronic devices, the simulation, and output a plurality of pieces of first data representing outputs by the plurality of electronic devices (performing simulation on a simulation model comprising of a plurality of electronic device such as sensors, controller, Zhang teaches the first data sample set being to actual measurement value and second data sample set being simulation value determined by the simulation model, so the named order as taught by Zhang is opposite to that in the claim, Two Paragraphs from bottom at Page 5 of ZHANG; Last Two Paragraphs from bottom of Page 8 and Page 9 of ZHANG); and estimate, based on the first data and a plurality of pieces of second data representing outputs obtained by operating the plurality of electronic devices, mapping data representing a correspondence between the plurality of electronic devices that output the first data and the plurality of electronic devices that output the second data (a second data sample set being to actual measurement value of the plurality of electronic devices, Last Para. of Page 8 to Para. [0002] of Page 9 of ZHANG; See also Last Para. of Page 9 through Para. [0004] of Page 10 of ZHANG teaching determining the second data sample set according, corresponding to the simulation outputs, to the first data sample set, corresponding to the actual measured outputs, and the preset simulation model to determine a set of training data sample; these teachings in combination under BRI read onto the estimate … mapping data as recited in this limitation).
ZHANG does not appear to explicitly disclose all of wherein the second data is given second metadata, and the one or more processors are further configured to: output the first data to which first metadata is given, the first metadata being determined independently of the second metadata: and estimate the mapping data based on a matching degree between the first metadata and the second metadata.
SHEU, however, is in the same field of generating fused sensor data by associating metadata corresponding to first sensor data with metadata corresponding to second sensor data. (Para. [0001] of SHEU) and teaches perform, by using a model for performing a simulation of operations of a plurality of electronic devices, the simulation (a virtual sensor system 624 that represents a virtual simulation of the operation of one or more of the sensors 604 … the virtual sensor system 624 may, in turn, include one or more engines, program modules, components, or the like including, without limitation, a predictive model 626 and a training/calibration engine 628 … each of the engines/components depicted in FIG. 6 may include logic for performing any of the processes or tasks described earlier in connection with correspondingly named engines/components, Para. [0077] of SHEU), and output a plurality of pieces of first data representing outputs by the plurality of electronic devices (first sensor data captured by a first sensor, Para. [0041]; See also a set of synchronized frames is obtained [output], where each synchronized frame corresponds to a particular image frame of the 2D image data [second sensor data] and a particular data frame of the 3D point cloud data [first sensor data] and is associated with a set of labels [metadata] assigned to the particular image frame [second sensor data] and a set of labels [metadata] assigned to the particular data frame of the 3D point cloud data [first sensor data] … then, a metadata association algorithm, or more specifically, a label association algorithm may be executed on the set of synchronized frames on a frame-by-frame basis to fuse the 2D image data and the 3D point cloud data, Para. [0039] of SHEU); and estimate, based on the first data and a plurality of pieces of second data representing outputs obtained by operating the plurality of electronic devices, mapping data representing a correspondence between the plurality of electronic devices that output the first data and the plurality of electronic devices that output the second data (performing a frame synchronization of the labeled 3D point cloud data [first sensor data] and the labeled 2D image data [second sensor data] to obtain a set of synchronized frames, where each synchronized frame corresponds to a frame of the 2D image data and a corresponding frame of the 3D point cloud data when the camera and the LiDAR are capturing the same portion of a scene … because the LiDAR and the camera occupy different positions on the vehicle and each have a different field-of-view (FOV) and a different data frame capture rate, a set of extrinsics may be calibrated for the LiDAR and the camera and used along with timing data from the LiDAR and camera to perform the frame synchronization … the calibrated set of extrinsics may include rotational and translational information that, along with timing data, allows for data frames captured by the LiDAR to be aligned with image frames captured by the camera to ensure that a LiDAR data frame and an image frame, when synchronized, correspond to data captured from the same (or substantially the same) portion of a scene over the same (or substantially the same) period of time, Para. [0038] of SHEU; [the same or substantially same is interpreted as best estimate]; See also matching pairs of 2D and 3D labels may be identified by selecting, for each 2D label, the 3D label with which the 2D label has the highest similarity score or vice versa … in some example embodiments, a matching pair may only be identified if the corresponding similarity score of the matching pair satisfies (e.g., meets or exceeds) a threshold score, Para. [0040] of SHEU; See also each similarity score may be a quantitative value indicative of a likelihood that a 2D bounding box and a 3D bounding box to which the similarity score corresponds represents the same object, Para. [0066] of SHEU), wherein the second data is given second metadata (second metadata may be associated with the 2D image data, Para. [0036] of SHEU), and the one or more processors are further configured to: output the first data to which first metadata is given (associating first metadata (e.g., labels) assigned to first sensor data captured by a first sensor, Para. [0041] of SHEU), the first metadata being determined independently of the second metadata (metadata is associated separately and at different times with sensor data from different sensors, Para. [0033] of SHEU; See also at block 410 of the method 400, the metadata assignment engine 216 may be executed to assign labels [first metadata] to the 3D point cloud data 212 [first sensor data] to obtain labeled 3D point cloud data 218 … at block 412 of the method 400, the metadata assignment engine 216 may be executed to assign labels [second metadata] to the 2D image data 214 [second sensor data] to obtain labeled 2D image data 220, Para. [0060] of SHEU); and estimate the mapping data based on a matching degree between the first metadata and the second metadata (matching pairs of 2D and 3D labels may be identified by selecting, for each 2D label, the 3D label with which the 2D label has the highest similarity score or vice versa … in some example embodiments, a matching pair may only be identified if the corresponding similarity score of the matching pair satisfies (e.g., meets or exceeds) a threshold score, Para. [0040] of SHEU; [the similarity score for identifying the matching pair is interpreted as corresponding to a matching degree]).
It would have been obvious for one of ordinary skill in the art before the effective filing date of the invention to modify the simulation methods of ZHANG with the metadata-based sensor data fusing of SHEU for the purpose of establishing a correspondence between sensor data from different sensors (Para. [0004] of SHEU).
Regarding claim 3, ZHANG as modified by SHEU discloses the device according to claim 1 (as shown above), wherein the one or more processors are further configured to: extract first feature data representing features of the plurality of pieces of first data and second feature data representing features of the plurality of pieces of second data (object detection processing can be executed on the captured images to recognize, interpret, and/or identify objects in the images and/or visual cues of the objects, Para. [0031] of SHEU; [detecting objects in the first and second sensor/image data is interpreted as corresponding to extracting features]; See also LiDARs can be utilized to detect objects (e.g., other vehicles, road signs, pedestrians, buildings, etc.) in an environment around a vehicle. LiDARs can also be utilized to determine relative distances between objects in the environment and between objects and the vehicle, Para. [0031] of SHUE; See also associating the first metadata with the first sensor data to obtain the labeled first sensor data includes associating a 3D bounding box with a first object present in the 3D point cloud data, and associating the second metadata with the second sensor data to obtain the labeled second sensor data includes associating a 2D bounding box with a second object present in the 2D image data, Para. [0007] of SHEU); and estimate the mapping data based on the first feature data and the second feature data (each synchronized frame is associated with i) a respective set of 3D labels [mapping data] representative of a respective set of objects [first feature data] present in the respective portion of the 3D point cloud data and ii) a respective set of 2D labels [mapping data] representative of a respective set of objects [second feature data] present in the respective portion of the 2D image data, Para. [0009] of SHEU).
Regarding claim 4, ZHANG as modified by SHEU discloses the device according to claim 3 (as shown above), wherein the one or more processors are configured to extract the second feature data by analyzing the second metadata given to the second data (each label [second metadata] applied to the 2D image data may include a 2D bounding box formed around a corresponding object identified in the 2D data as well as an indication of the type of object [second feature data] … in some example embodiments, the object type identifier [second feature data] may be a code or the like that identifies the type of object, Para. [0036] of SHEU).
Regarding claim 5, ZHANG as modified by SHEU discloses the device according to claim 1 (as shown above), wherein the one or more processors are configured to determine a condition of the simulation based on mapping data obtained in advance (obtaining physical state of the device to be controlled and inputting the detected state parameter into the preset model for simulation; the physical state of the device is a condition of the simulation, and the physical state is part of the mapping data obtained in advance, 2nd to last Para. of Page 8 of ZHANG; See also various metadata may need to be associated with the sensor data prior to the sensor data becoming usable as a training dataset … such metadata may include, for example, labels that are assigned to/associated with the sensor data … the labels may include indicia that identify the locations of various objects present in the sensor data … additionally, the labels may identify object types of the objects … in example scenarios, the labels may be manually associated with the sensor data to form the training dataset, Para. [0032] of SHEU; [the manual labeling prior to training is interpreted as a condition/criteria of the simulation based on the mapping [label] data obtained in advance]).
Regarding claim 6, ZHANG as modified by SHEU discloses the device according to claim 1 (as shown above), wherein the one or more processors are configured to determine a condition of the simulation based on the second data obtained in advance (obtaining physical state of the device to be controlled and inputting the detected state parameter into the preset model for simulation; the physical state of the device is a condition of the simulation, and the physical state is part of the second data obtained in advance, 2nd to last Para. of Page 8 of ZHANG; See also various metadata may need to be associated with the sensor data prior to the sensor data becoming usable as a training dataset … such metadata may include, for example, labels that are assigned to/associated with the sensor data … the labels may include indicia that identify the locations of various objects present in the sensor data … additionally, the labels may identify object types of the objects … in example scenarios, the labels may be manually associated with the sensor data to form the training dataset, Para. [0032] of SHEU; [the manual labeling of sensor data prior to training is interpreted as a condition/criteria of the simulation based on the sensor data [including the second sensor data] obtained in advance]).
Regarding claim 7, ZHANG as modified by SHEU discloses the device according to claim 1 (as shown above), wherein the one or more processors are configured to: determine whether or not estimation of the mapping data is ended (ZHANG teaches determining if the search for the next action and time should continue or end based on obtained the reward and state of the safety score, Last Para. of Page 10 of ZHANG; [this ZHANG teaching is interpreted as corresponding to this limitation]); and when the estimation of the mapping data is not ended, determine a condition of the simulation and further perform the simulation according to the determined condition (2nd to Last Para. on Page 8 and Last Para. of Page 10 of ZHANG teaches obtaining physical state of the device to be controlled and inputting the detected state parameter into the preset model for simulation; the physical state of the device is a condition of the simulation; Zhang teaches selecting a group of data as a track of the starting point, … re-inputting the simulation model to obtain results; if results are reliable continue with the next search; [this ZHANG teaching is interpreted as meaning repeating the step of simulation according to the determined condition and checking if results are reliable or not, which are interpreted as corresponding to the estimation of the mapping data is ended or not]).
Regarding claim 8, ZHANG as modified by SHEU discloses the device according to claim 1, wherein the one or more processors are further configured to: calculate, by using a learning model learned to output a matching degree between the second data and the first data, the matching degree, and estimate the mapping data by using the calculated matching degree (fused 2D and 3D sensor data that is indicative of matching 2D and 3D labels provides a richer training dataset for use in training a machine learning model, and thus, results in an improved learning model that is capable of performing its designated classification task across a broader range of potential types of objects and/or with improved accuracy, Para. [0041] of SHEU; See also ZHANG teaches using real offline data training joint condition probability distribution model may be according to the first data sample set, corresponding to the measured second data as claimed, and the preset simulation model to determine the second data sample set, corresponding to the simulation first data set as claimed, Para. [0001] of Page 11 of ZHANG; [this ZHANG teaching of performing joint condition probability between the two the claimed second data and first data to determine the data sample set is interpreted as corresponding to this limitation]); output the estimated mapping data (ZHANG teaches determining the data sample set from the calculation as discussed in the above limitation; the determined data sample set is interpreted as the estimated mapping data output, Para. [0001] of Page 11 of ZHANG); and learn the learning model by using correct answer data of the mapping data set by referring to the outputted mapping data (ZHANG teaches using data training for the joint condition probability distribution model, which is a deep neural network model, learning model; this teaching means the learning model learns using the correct answer data of the mapping data set by referring to the outputted mapping data, Para. [0001] of Page 11 of ZHANG).
Regarding claim 9, ZHANG as modified by SHEU discloses the device according to claim 1, wherein the one or more processors are configured to: calculate, for each of a plurality of combinations each of which indicates a candidate of a correspondence between the plurality of pieces of first data and the plurality of pieces of second data, an error between first correlation data representing a correlation between the plurality of pieces of first data and second correlation data representing a correlation between the plurality of pieces of second data (matching pairs of 2D and 3D labels may be identified by selecting, for each 2D label, the 3D label with which the 2D label has the highest similarity score or vice versa. In some example embodiments, a matching pair may only be identified if the corresponding similarity score of the matching pair satisfies (e.g., meets or exceeds) a threshold score … thus, in some example embodiments, one of more 2D labels may remain unmatched if there is no 3D label with which the 2D label has a similarity score that satisfies the threshold score … the converse is true as well … that is, one or more 3D labels may remain unmatched if there is no 2D label with which the 3D label has a similarity score that satisfies the threshold score, Para. [0040] of SHEU; [when the similarity score meets a threshold, it is interpreted as no error, and when the similarity score does not meet the threshold, it is interpreted as error]), and estimate a correspondence indicated by one of the plurality of combinations having the error smaller than the error calculated for other combinations, as the mapping data based on the calculated error (matching pairs of 2D and 3D labels may be identified by selecting, for each 2D label, the 3D label with which the 2D label has the highest similarity score or vice versa. In some example embodiments, a matching pair may only be identified if the corresponding similarity score of the matching pair satisfies (e.g., meets or exceeds) a threshold score … thus, in some example embodiments, one of more 2D labels may remain unmatched if there is no 3D label with which the 2D label has a similarity score that satisfies the threshold score … the converse is true as well … that is, one or more 3D labels may remain unmatched if there is no 2D label with which the 3D label has a similarity score that satisfies the threshold score, Para. [0040] of SHEU; [when the similarity score meets a threshold, it is interpreted as no error, and when the similarity score does not meet the threshold, it is interpreted as error]).
Regarding claim 10, ZHANG as modified by SHEU discloses the device according to claim 1, wherein the model is a physical model for performing the simulation (2nd to Last Para. of Page 8 and Para. [0001] of Page 9 of ZHANG teaches performing obtaining physical state of the device to be controlled and inputting the detected state parameter into the preset model for simulation; the state parameter is used for characterizing the physical state of an automobile, for example, current vehicle speed; hence, the preset model for simulation as taught is as physical model).
Regarding claim 11, ZHANG as modified by SHEU discloses the device according to claim 1, wherein the one or more processors are configured to perform the simulation and output the plurality of pieces of first data representing the outputs by the plurality of electronic devices (Last two Paras. of Page 8 thru Para. [0002] of Page 9 of ZHANG teaches performing simulation on a simulation model comprising of a plurality of electronic device such as sensors, controller, to produce simulation value; See also SHEU teaches a virtual sensor system 624 that represents a virtual simulation of the operation of one or more of the sensors 604 … the virtual sensor system 624 may, in turn, include one or more engines, program modules, components, or the like including, without limitation, a predictive model 626 and a training/calibration engine 628 … each of the engines/components depicted in FIG. 6 may include logic for performing any of the processes or tasks described earlier in connection with correspondingly named engines/components, Para. [0077] of SHEU).
Claim 12 has substantially similar limitations as recited in device claim 1 in terms of a method; therefore, it is rejected under 35 U.S.C. § 103, mutatis mutandis, for the same reasons.
Claim 13 has substantially similar limitations as recited in device claim 1 in terms of a non-transitory computer-readable medium; therefore, it is rejected under 35 U.S.C. § 103, mutatis mutandis, for the same reasons. See also computer program stored in readable storage medium, 3rd to Last Para. of Page 15 of ZHANG.
Claim 14 has substantially similar limitations as recited in device claim 1 in terms of a system; therefore, it is rejected under 35 U.S.C. § 103, mutatis mutandis, for the same reasons. See also a myriad of sensors (plurality of electronic devices), Para. [0031] of SHEU.
Claim 15 is rejected under 35 U.S.C. § 103 as being unpatentable over ZHANG et al. (CN 113743603), hereinafter “ZHANG”, which refers to the English language translation of ZHANG et al., in view of SHEU et al. (U.S. Patent Application Publication No. 2022/0027684 A1), and further in view of ABBEY et al. (U.S. Patent Application Publication No. 2021/0173969 A1).
Regarding claim 15, ZHANG as modified by SHEU discloses the device according to claim 1 (as shown above) but appears to fail to explicitly disclose wherein the first metadata includes a name of one of the plurality of electronic devices that outputs a corresponding piece of first data, the second metadata includes a name of one of the plurality of electronic devices that outputs a corresponding piece of second data, and the name included in the first metadata is determined independently of the name included in the second metadata.
ABBEY, however, teaches wherein the first metadata includes a name of one of the plurality of electronic devices that outputs a corresponding piece of first data, the second metadata includes a name of one of the plurality of electronic devices that outputs a corresponding piece of second data (BMS 11 may provide a listing of controllers for building 10 (e.g., as part of a network of data points) that have the physical location (e.g., room name) of the controller in the name of the controller itself. Building object creation module 152 may extract room names from the names of BMS controllers defined in the network of data points and create building objects for each extracted room, Para. [0080] of ABBEY; See also the representation of the digital twins may include the ability to simulate the dynamics of the real-world environment … this may include simulating a device's physical response to operation, such as heating through friction or material fatigue, Para. [0131] of ABBEY; See also the models that are used to simulate the digital twins are modified when the simulated predictions do not match the information provided by the physical counterparts, Para. [0131] of ABBEY; [the simulated predictions and/or simulated physical response are interpreted as corresponding to the output]), and the name included in the first metadata is determined independently of the name included in the second metadata (equipment definition module 154 may generate the abstracted text string “SUP-FLOW” from the equipment-specific data point “VMA-20.SUP-FLOW.” Advantageously, the abstracted text string matches other equipment-specific data points corresponding to the supply air flow rates of other BMS devices (e.g., “VMA-18.SUP-FLOW,” “SUP-FLOW.VMA-01,” etc.) … equipment definition module 154 may store a name, label, and/or search criteria for each point definition in memory 138, Para. [0086] of ABBEY).
It would have been obvious for one of ordinary skill in the art before the effective filing date of the invention to modify the simulation methods of ZHANG as modified by SHEU with the equipment-specific naming/labeling of ABBEY for the purpose of generating a user-friendly/searchable label for equipment (Paras. [0084] & [0085] of ABBEY).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure: WHITLEY et al. (U.S. Patent Application Publication No. 2020/0272640 A1) discloses an information processing device comprising: one or more processors (a method implemented by one or more processors, Para. [0002] of WHITLEY) configured to: perform, by using a model for performing a simulation of operations of a plurality of electronic devices, the simulation (simulation workflow can identify specific uses, which can be embodied in a number of models … models can include company specific models, equipment health data, equipment monitoring data, Para. [0038] of WHITLEY).
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JOHN P HOCKER whose telephone number is (571)272-0501. The examiner can normally be reached Monday-Friday 9:00 AM - 5:00 PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Rehana Perveen can be reached on (571)272-3676. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JOHN P HOCKER/Examiner, Art Unit 2189
/REHANA PERVEEN/Supervisory Patent Examiner, Art Unit 2189
Prosecution Insights