DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Applicant’s submission filed on 02/27/2026 has been entered. Claims 1, 15, and 18 were amended. Claims 24-26 were added. Claims 1-26 are pending in the application.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-2, 12-15, 18, and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Harp et al. (US 2015/0186418) in view of Bhushan et al. (US 12141426), Anadure et al. (US 10573019), and Baldwin et al. (US 2018/0367732).
Regarding claim 1, Harp teaches/suggests: A computer-implemented method, comprising:
capturing, by one or more of a plurality of sensors of a computing device, respective sensor data corresponding to a physical environment (Harp [0021] “structured light scanners may capture images of an object”);
generating, by a processor of the computing device (Harp Fig. 6: processor 610) from the respective sensor data, a multidimensional dataset representative of the physical environment, the multidimensional dataset (Harp [0027] “The database 106 may store all data sets for a 3D object data model in any number of various forms from raw data captured to processed data for display”) including:
the respective sensor data (Harp [0021] “structured light scanners may capture images of an object”);
geometric data corresponding with the physical environment (Harp [0021] “multiple scans of an object may be processed into a merged mesh and assets data model”); and
semantic data corresponding with the physical environment (Harp [0025] “for each object, the semantics and search index 114 may index or label components of the images”); and
identifying and indexing, by the processor of the computing device, a plurality of dimensions of the multidimensional dataset, such that the plurality of dimensions of the multidimensional dataset can be, at least one of, searched, queried, or modified (Harp [0025] “for each object, the semantics and search index 114 may index or label components of the images” [0030] “the search query results 204 may include data obtained from the data network 230 that pertains to an object indicated in the search query 202”),
Harp further teaches/suggests a plurality of objects in a physical environment (Harp [0019] “the visual search system may also identify one or more other objects in the environment of the object based on the information from the 3D model database”). Harp is silent regarding respective sensor data corresponding to a plurality of objects in a physical environment. Bhushan, however, teaches/suggests respective sensor data corresponding to a plurality of objects in a physical environment (Bhushan col. 63 ll. 8-20 “XR application 1814 may further receive a two-dimensional (2D) or three-dimensional (3D) model of the objects identified via the optical data markers. The 2D or 3D model may be a simple outline” col. 63 line 53 – col. 64 line 9 “Once the user completes the scan, XR application 1814 may generate a rough outline of the object”). Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to modify the capturing of Harp to include the other objects as taught/suggested by Bhushan for the indexing/labeling.
Harp as modified by Bhushan does not teach/suggest provide, based on the search, query or modification, a visualization of the physical environment on a display device. Anadure, however, teaches/suggests provide, based on the search, query or modification, a visualization of the physical environment on a display device (Anadure col. 10 ll. 1-21 “the search result 222 is displayed as augmented reality digital content with the user interface 120 as part of the live stream of digital images 112”). Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to modify the search query results of Harp as modified by Bhushan to be displayed as taught/suggested by Anadure for augmented reality.
Harp as modified by Bhushan and Anadure does not teach/suggest:
wherein capturing the respective sensor data includes providing capturing guidance by inferring an intent of a user based on a movement of the computing device, and providing the capturing guidance to the user, based on the intent of the user.
Baldwin, however, teaches/suggests providing capturing guidance (Baldwin [0021] “approaches provide for a number of visual, audio, and/or haptic cues to be presented to the user to guide the user in panning the device”). Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to modify the capturing of Harp as modified by Bhushan and Anadure to include the cues of Baldwin to guide the user. As such, Harp as modified by Bhushan, Anadure, and Baldwin teaches/suggests:
providing capturing guidance by inferring an intent of a user based on a movement of the computing device, and providing the capturing guidance to the user, based on the intent of the user (Bhushan col. 63 line 34 – col. 64 line 9 “a user may point camera 1820 of client device 404 at a physical object of interest … Once the user completes the scan, XR application 1814 may generate a rough outline of the object” Baldwin [0021] “approaches provide for a number of visual, audio, and/or haptic cues to be presented to the user to guide the user in panning the device”).
Regarding claim 2, Harp as modified by Bhushan, Anadure, and Baldwin teaches/suggests: The computer-implemented method of claim 1, wherein:
the geometric data is derived from the respective sensor data (Harp [0021] “multiple scans of an object may be processed into a merged mesh and assets data model”); and
the semantic data is determined from the geometric data and the respective sensor data (Harp [0025] “for each object, the semantics and search index 114 may index or label components of the images (e.g., per pixel) as having a certain texture, color, shape, geometry, attribute, etc.”).
Regarding claim 12, Harp as modified by Bhushan, Anadure, and Baldwin teaches/suggests: The computer-implemented method of claim 1, wherein the plurality of dimensions include one or more of:
lighting of the physical environment;
objects in the physical environment (Harp [0019] “the visual search system may also identify one or more other objects in the environment of the object based on the information from the 3D model database”);
a time corresponding with capturing the respective sensor data;
regions of the physical environment;
physical dimensions of the physical environment; or
surfaces in the physical environment.
Regarding claim 13, Harp as modified by Bhushan, Anadure, and Baldwin teaches/suggests: The computer-implemented method of claim 1, wherein the plurality of sensors include one or more of:
an image sensor (Harp [0021] “structured light scanners may capture images of an object”);
a depth sensor;
a location sensor;
an orientation sensor;
a motion sensor;
a light sensor;
a pressure sensor; or
a temperature sensor.
Regarding claim 14, Harp as modified by Bhushan, Anadure, and Baldwin teaches/suggests: The computer-implemented method of claim 1, further comprising:
receiving, at the computing device or another computing device, a visualization request including a query identifying a dimension of the plurality of dimensions of the multidimensional dataset (Harp [0035] “the search module 220 may be configured to receive the search query 202 … include an image of a box with features on a given surface”); and
providing, on the display device of the computing device or the other computing device, the visualization based on the multidimensional dataset and the query (Harp [0035] “the search module 220 may associate the features with a cereal box based on shape-matching the features with corresponding shapes in the semantic database 222 to generate the search query results 204 including images of the cereal box obtained from the data network 230” Anadure col. 10 ll. 1-21 “the search result 222 is displayed as augmented reality digital content with the user interface 120 as part of the live stream of digital images 112”).
The same rationale to combine as set forth in the rejection of claim 1 is incorporated herein.
Claim 15 recites limitation(s) similar in scope to those of claim 1, and is rejected for the same reason(s). Harp as modified by Bhushan, Anadure, and Baldwin further teaches/suggests a memory having instructions stored thereon (Harp Fig. 6: system memory 620).
Claims 18 and 20 recite limitation(s) similar in scope to those of claims 1 and 14, respectively, and are rejected for the same reason(s). Harp as modified by Bhushan, Anadure, and Baldwin further teaches/suggests a non-transitory computer-readable medium having instructions stored thereon (Harp Fig. 6: system memory 620).
Claim(s) 3 and 4 is/are rejected under 35 U.S.C. 103 as being unpatentable over Harp et al. (US 2015/0186418) in view of Bhushan et al. (US 12141426), Anadure et al. (US 10573019), and Baldwin et al. (US 2018/0367732) as applied to claim 2 above, and further in view of McGuinness et al. (US 2022/0245715).
Regarding claim 3, Harp as modified by Bhushan, Anadure, and Baldwin does not teach/suggest: The computer-implemented method of claim 2, wherein the semantic data is further determined based on features of the physical environment identified by an augmented reality (AR) framework. McGuinness, however, teaches/suggests the semantic data is further determined based on features of the physical environment identified by an augmented reality (AR) framework (McGuinness [0050] “the AR framework available for iOS facilitates the measurement of objects”). Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to modify the computing device of Harp as modified by Bhushan, Anadure, and Baldwin to include the AR framework of McGuinness to facility the indexing/labeling.
Regarding claim 4, Harp as modified by Bhushan, Anadure, Baldwin, and McGuinness teaches/suggests: The computer-implemented method of claim 3, wherein the AR framework is implemented by one of the processor of the computing device, or another computing device operatively coupled with the computing device (McGuinness [0050] “the AR framework available for iOS facilitates the measurement of objects”). The same rationale to combine as set forth in the rejection of claim 3 is incorporated herein.
Claim(s) 5-7, 9-11, 16-17, and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Harp et al. (US 2015/0186418) in view of Bhushan et al. (US 12141426), Anadure et al. (US 10573019), and Baldwin et al. (US 2018/0367732) as applied to claims 1, 15, and 18 above, and further in view of Newhouse et al. (US 2015/0243085).
Regarding claim 5, Harp as modified by Bhushan, Anadure, and Baldwin teaches/suggests: The computer-implemented method of claim 1, wherein the respective sensor data is first sensor data (Harp [0021] “structured light scanners may capture images of an object”). Harp as modified by Bhushan, Anadure, and Baldwin does not teach/suggest the computer-implemented method further comprising:
capturing second sensor data corresponding to the physical environment; and
modifying, based on the second sensor data, the multidimensional dataset.
Newhouse, in view of Harp, teaches/suggests:
capturing second sensor data corresponding to the physical environment (Newhouse [0019] “Dynamic content can be captured by device”); and
modifying, based on the second sensor data, the multidimensional dataset (Harp [0027] “The database 106 may store all data sets for a 3D object data model in any number of various forms from raw data captured to processed data for display” Newhouse [0019] “combined with a preexisting or simultaneously captured VAR scene”).
Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to modify the capturing of Harp as modified by Bhushan, Anadure, and Baldwin to capture again as taught/suggested by Newhouse to capture dynamic content.
Regarding claim 6, Harp as modified by Bhushan, Anadure, Baldwin, and Newhouse teaches/suggests: The computer-implemented method of claim 5, wherein capturing the second sensor data includes providing capturing guidance, via the computing device, the capturing guidance being based on the multidimensional dataset (Harp [0027] “The database 106 may store all data sets for a 3D object data model in any number of various forms from raw data captured to processed data for display” Baldwin [0021] “approaches provide for a number of visual, audio, and/or haptic cues to be presented to the user to guide the user in panning the device” Newhouse [0019] “Dynamic content can be captured by device”). The same rationales to combine as set forth in the rejection of claims 1 and 5 are incorporated herein.
Regarding claim 7, Harp as modified by Bhushan, Anadure, Baldwin, and Newhouse teaches/suggests: The computer-implemented method of claim 5, wherein the geometric data is first geometric data (Harp [0021] “multiple scans of an object may be processed into a merged mesh and assets data model”), and the sematic data is first semantic data (Harp [0025] “for each object, the semantics and search index 114 may index or label components of the images”), the modifying the multidimensional dataset including:
adding the second sensor data to the multidimensional dataset (Harp [0027] “The database 106 may store all data sets for a 3D object data model in any number of various forms from raw data captured to processed data for display” Newhouse [0019] “combined with a preexisting or simultaneously captured VAR scene”);
adding second geometric data corresponding with the physical environment to the multidimensional dataset (Harp [0021] “multiple scans of an object may be processed into a merged mesh and assets data model” Newhouse [0019] “combined with a preexisting or simultaneously captured VAR scene”); and
adding second semantic data corresponding with the physical environment to the multidimensional dataset (Harp [0025] “for each object, the semantics and search index 114 may index or label components of the images” Newhouse [0019] “combined with a preexisting or simultaneously captured VAR scene”).
The same rationale to combine as set forth in the rejection of claim 5 is incorporated herein.
Regarding claim 9, Harp as modified by Bhushan, Anadure, Baldwin, and Newhouse teaches/suggests: The computer-implemented method of claim 5, wherein modifying the multidimensional dataset includes:
aggregating the second sensor data with the first sensor data to produce aggregated sensor data (Harp [0021] “structured light scanners may capture images of an object” Newhouse [0019] “combined with a preexisting or simultaneously captured VAR scene”);
modifying the geometric data based on the aggregated sensor data (Harp [0021] “multiple scans of an object may be processed into a merged mesh and assets data model” Newhouse [0019] “combined with a preexisting or simultaneously captured VAR scene”); and
modifying the semantic data based on the aggregated sensor data and the modified geometric data (Harp [0025] “for each object, the semantics and search index 114 may index or label components of the images” Newhouse [0019] “combined with a preexisting or simultaneously captured VAR scene”).
The same rationale to combine as set forth in the rejection of claim 5 is incorporated herein.
Regarding claim 10, Harp as modified by Bhushan, Anadure, Baldwin, and Newhouse teaches/suggests: The computer-implemented method of claim 5, wherein capturing the second sensor data includes capturing the second sensor data with one or more of the plurality of sensors of the computing device (Harp [0021] “structured light scanners may capture images of an object” Newhouse [0019] “Dynamic content can be captured by device”). The same rationale to combine as set forth in the rejection of claim 5 is incorporated herein.
Regarding claim 11, Harp as modified by Bhushan, Anadure, Baldwin, and Newhouse teaches/suggests: The computer-implemented method of claim 5, wherein, the computing device is a first computing device; and the capturing second sensor data includes capturing respective sensor data for a second plurality of sensors of a second computing device (Newhouse [0021] “one or more user devices 100A, 100B, 100C, can be used to capture virtual or augmented reality (VAR) scenes 102A, 102B, 102C”). The same rationale to combine as set forth in the rejection of claim 5 is incorporated herein.
Claims 16 and 19 recite limitation(s) similar in scope to those of claim 5, and are rejected for the same reason(s).
Claim 17 recites limitation(s) similar in scope to those of claim 6, and is rejected for the same reason(s).
Claim(s) 8 is/are rejected under 35 U.S.C. 103 as being unpatentable over Harp et al. (US 2015/0186418) in view of Bhushan et al. (US 12141426), Anadure et al. (US 10573019), Baldwin et al. (US 2018/0367732), and Newhouse et al. (US 2015/0243085) as applied to claim 7 above, and further in view of Atanassov et al. (US 2010/0157079).
Regarding claim 8, Harp as modified by Bhushan, Anadure, Baldwin, and Newhouse does not teach/suggest: The computer-implemented method of claim 7, wherein modifying the multidimensional dataset further includes at least one of:
determining differences between the first sensor data and the second sensor data;
determining differences between the first geometric data and the second geometric data; or
determining differences between the first semantic data and the second semantic data.
Atanassov, however, teaches/suggests:
determining differences between the first sensor data and the second sensor data (Atanassov [0051] “When the MS difference exceeds the threshold value for a particular macro block, the particular macro block is determined to be too different between the first image data 406 and the aligned image data for the second image, and thus the image data should not be combined for the particular macro block”);
Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to modify the images of Harp as modified by Bhushan, Anadure, Baldwin, and Newhouse such that their differences are determined as taught/suggested by Atanassov to determine if they’re too different to be combined.
Claim(s) 21 is/are rejected under 35 U.S.C. 103 as being unpatentable over Harp et al. (US 2015/0186418) in view of Bhushan et al. (US 12141426), Anadure et al. (US 10573019), and Baldwin et al. (US 2018/0367732) as applied to claim 1 above, and further in view of Givon (US 2013/0120319).
Regarding claim 21, Harp, Bhushan, Anadure, and Baldwin are silent regarding: The computer-implemented method of claim 1, wherein the multidimensional dataset includes at least one of a time dimension and a dimension related to a distance between two of the plurality of objects. Givon, however, teaches/suggests a time dimension (Givon [0103] “produced based on images acquired at different times, to generate a multidimensional image data set (steps 4000B and 4000C) which may include time as one of the dimensions”). Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to modify the capturing of Harp as modified by Bhushan, Anadure, and Baldwin to include the time dimension of Givon for the indexing/labeling.
Claim(s) 22-24 is/are rejected under 35 U.S.C. 103 as being unpatentable over Harp et al. (US 2015/0186418) in view of Bhushan et al. (US 12141426), Anadure et al. (US 10573019), and Baldwin et al. (US 2018/0367732) as applied to claim 1 above, and further in view of Grimm et al. (US 2018/0061077).
Regarding claim 22, Harp as modified by Bhushan, Anadure, and Baldwin does not teach/suggest: The computer-implemented method of claim 1, wherein modification of the plurality of dimensions of the multidimensional dataset includes removing at least one of the plurality of dimensions of the multidimensional dataset. Grimm, however, teaches/suggests modification of the plurality of dimensions of the multidimensional dataset includes removing at least one of the plurality of dimensions of the multidimensional dataset (Grimm [0142] “The data vector for each spatial point could be mapped onto a new basis in which the greatest variation is aggregated in the first dimensions. This also provides the option of usefully reducing the dimensionality by discarding the higher dimensions”). Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to modify the multidimensional dataset of Harp as modified by Bhushan, Anadure, and Baldwin such that a dimension is discarded as taught/suggested by Grimm to reduce the dimensionality.
Regarding claim 23, Harp as modified by Bhushan, Anadure, Baldwin, and Grimm teaches/suggests: The computer-implemented method of claim 22, wherein removing the at least one of the plurality of dimensions of the multidimensional dataset includes aggregating values for a dimension (Grimm [0142] “The data vector for each spatial point could be mapped onto a new basis in which the greatest variation is aggregated in the first dimensions”). The same rationale to combine as set forth in the rejection of claim 22 is incorporated herein.
Regarding claim 24, Harp as modified by Bhushan, Anadure, Baldwin, and Grimm teaches/suggests: The computer-implemented method of claim 22, wherein removing the at least one of the plurality of dimensions of the multidimensional dataset includes removing a dimension related to lighting (Harp [0017] “the visual search system may obtain information associated with the 3D model such as … lighting conditions in the environment” Grimm [0142] “The data vector for each spatial point could be mapped onto a new basis in which the greatest variation is aggregated in the first dimensions. This also provides the option of usefully reducing the dimensionality by discarding the higher dimensions”). The same rationale to combine as set forth in the rejection of claim 22 is incorporated herein.
Claim(s) 25 is/are rejected under 35 U.S.C. 103 as being unpatentable over Harp et al. (US 2015/0186418) in view of Bhushan et al. (US 12141426), Anadure et al. (US 10573019), and Baldwin et al. (US 2018/0367732) as applied to claim 1 above, and further in view of Ravada et al. (US 2009/0091568).
Regarding claim 25, Harp as modified by Bhushan, Anadure, and Baldwin does not teach/suggest: The computer-implemented method of claim 1, wherein the multidimensional dataset includes a dimension related to a distance between two of the plurality of objects, and the dimension related to the distance is indexed and can be used to respond to a query related to the distance. Ravada teaches/suggests a dimension related to a distance between two of the plurality of objects, and the dimension related to the distance is indexed and can be used to respond to a query related to the distance (Ravada [0047] “Given a geometry as the query object and a distance d, a within distance query may identify geometries that are within distance d of the query object”). Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to modify the multidimensional dataset of Harp as modified by Bhushan, Anadure, and Baldwin to include the distance of Ravada for a within distance query.
Claim(s) 26 is/are rejected under 35 U.S.C. 103 as being unpatentable over Harp et al. (US 2015/0186418) in view of Bhushan et al. (US 12141426), Anadure et al. (US 10573019), and Baldwin et al. (US 2018/0367732) as applied to claim 1 above, and further in view of Yalniz et al. (US 2015/0363943).
Regarding claim 26, Harp as modified by Bhushan, Anadure, and Baldwin does not teach/suggest: The computer-implemented method of claim 1, wherein the multidimensional dataset is configured to be searched against a query for a spatial recommendation related to the physical environment. Yalniz, however, teaches/suggests the multidimensional dataset is configured to be searched against a query for a spatial recommendation related to the physical environment (Yalniz [0047] “the chaise lounge chair (recommended item 296a) may be recommended based on the third empty region 406c being within a threshold distance to the first object 403a (e.g., the couch) which the image analysis engine 215 may identify based upon its shape or pattern” [0059] “the recommendation service 218 may employ various design patterns stored in the data store 212 to identify items to recommend to the user”). Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to modify the indexing/labeling of Harp as modified by Bhushan, Anadure, and Baldwin to be queried as taught/suggested by Yalniz to identify items to recommend to the user.
Response to Arguments
Applicant's argument(s) filed on 02/27/2026 have been fully considered but they are moot in view of the new ground(s) of rejection set forth in this Office action.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
US 2004/0071368 – content-based querying
US 2016/0086257 – sales through interactive digital content
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a). A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANH-TUAN V NGUYEN whose telephone number is 571-270-7513. The examiner can normally be reached on M-F 9AM-5PM ET. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, JASON CHAN can be reached on 571-272-3022. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANH-TUAN V NGUYEN/
Primary Examiner, Art Unit 2619
Prosecution Insights