DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 112
Claim 26 is rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention. Claim 26 recites,
“... identifying pixels which represent the construction components within the panoramic view; and
“using the feature classifiers, identifying, by the AR device, construction features within the panoramic view...”
The specification discloses, in [0096], “In some embodiments, the application 122 includes a classification module. The server device 160 may retrieve and provide to the classification module feature classifiers containing several characteristics used to identify pixels which represent construction features, construction elements or construction components. Based on the feature classifiers, the classification module may identify construction features and remove superfluous portions of an image (i.e., that do not relate to the identified construction features). Various classification systems such as support vector machines, neural networks, naive Bayes, random forests, etc. may be utilised. The filtering process that removes portions of the image may be performed using a spatial heuristic algorithm, using machine learning techniques to compare pixels in the panoramic view to sample pixels, or in any other suitable manner.” The Specification discloses, comparing pixels in the panoramic view to sample pixels. The specification does not disclose identifying pixels which represent the construction components within the panoramic view or using the feature classifiers, identifying construction features within the panoramic view. These recitations of the panoramic view are considered to be NEW MATTER.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 26, 36: 27 and 28 is/are rejected under 35 U.S.C. 103 as being unpatentable over Rozenberg et al. U.S. Pub. No. 2018/0082414 in view of Connary et al. U.S. Patent No. 11,935,288, Weng CN 110 718 137 A and Trehan WO 2020/075185 A1.
Re: claim 26, Rozenberg teaches
26. (Currently Amended) A method of generating an augmented reality (AR) interface on an augmented reality (AR) device having at least one processor using a sensor of the AR device and machine learning (ML), comprising: receiving, by a virtual reality device or an augmented reality the AR device, a user selection for a construction region from a graphical user interface displayed by the AR device; (“The relevant construction site model... may be preselected by the user of the mobile device and likewise relayed to the server.”; Rozenberg, [0066], Fig. 1A)
The relevant construction site model indicates the user selected construction region.
(“The UI may allow for the user to provide his location with a given construction site by positioning himself over a 2D blueprint(s) of the construction site, retrieved by the system server from the 3D construction site models database and relayed to the mobile device. The user selection may, for example, be obtained through a screen touch of the user at the relevant position on the blueprint presented on a touchscreen display of the device and/or by his pointing of a cursor to the relevant position on the blueprint.”; Rozenberg, [0073], Fig. 2)
The user selects a relevant position on a blueprint presented on a touchscreen display (graphical user interface displayed) of the mobile computing device, where the selected position on the blueprint indicates a construction region.
(“An error indicator unit (e.g., an augmented reality rendering unit)... optionally and/or partially implemented on as-part-of a mobile computerized device, may indicate detected errors/differences between parallel objects and/or object sets, found within both the scene objects and the objects in the matching model.”; Rozenberg, [0090])
The mobile computerized device includes an augmented reality rendering unit, which makes the mobile computerized device an augmented reality device (AR device). Fig. 1A illustrates that the mobile computerized device includes a central processor (AR device having at least one processor).
retrieving, by the AR device, image data for one or more construction components that form part of the construction region as selected; (“One or more sensors, including a camera, of a computerized device may be utilized to digitize a scene of a construction site within which a user of the device is present. Digitized scene data and/or features and feature sets in the digitized scene may be extracted... Extracted features and/or construction associated objects derived therefrom, identified within one or more of the 3-dimensional construction site model may indicate the specific construction site within which the computerized device user is present, the location of the computerized device user within the site and the orientation of the computerized device at that location.”; Rozenberg, [0060])
Digitized scene data and features are extracted, where the extracted features include construction objects (construction components).
(“Digitized scene data (e.g., camera image, depth map) and the extracted feature characteristics... are relayed to the Vector Model Processor of the System Server.”; Rozenberg, [0066], Fig. 1)
The digitized scene data (image data) of the construction site (which includes one or more construction components that form part of the construction region as selected) is received (retrieved) from the camera and sent to the Server.
retrieving, by the AR device, construction information defining a construction requirement associated with one or more of the construction components; (“The scene digitizer, optionally and/or partially implemented on/as-part-of a mobile computerized device or appliance, may utilize one or more of the cameras and/or sensors to acquire a current digital representation/image of a real-world construction site scene as viewed from the specific position and at the specific angle of view, in which the scene digitizer is oriented. The feature detector may analyze the acquired digital image to recognize potential features (e.g., construction related features which are part of a construction object – for example, 4 corners of a window) within it. The extractor may extract from the image dimension and orientation related parameters associated with the detected features.”; Rozenberg, [0011], Fig. 1A)
Fig. 1A illustrates the scene digitizer of the mobile computerized device that acquires a digital image of the construction site scene, uses the feature detector to recognize construction related features of the construction objects (construction components), such as the four corners of a window and uses the feature extractor to extract (retrieve) dimensions of features (retrieving, by the AR device, construction information defining a construction requirement associated with one or more of the construction components), such as the window.
rendering, by an AR device, a first enhanced construction image in an AR environment shown in a display of the AR device that combines the retrieved image data for the construction components with a schematic representation of the construction information; (“On the mobile device, the shown Model Rendering and Error/Addition Indicator Unit... renders visual representation instructions for the augmentation of differences/deltas between the 3D model and the actual viewed/sensed construction site scene. Augmentation data of differences/deltas, or later construction stages differences/deltas, is relayed to the display/graphic processor of the mobile device and presented on the screen of the mobile device to the user as an overlay on: the 3D model view, the actual view being acquired by the camera of the mobile device and/or a combination of both.”; Rozenberg, [0070], Fig. 1A)
Fig. 1A illustrates a mobile rendering and error/addition indicator unit that renders a visual representation of the augmentation of differences between the 3D model (schematic representation of construction information) and the actual construction site scene (retrieved image data for the construction components), where this information is presented as an overlay on the actual view of the camera of the mobile device (rendering, by the AR device, a first enhanced construction image in an AR environment shown in a display of the AR device that combines the retrieved image data for construction components with a schematic representation of the construction information.
receiving, by the AR device, sensor data indicative of a location of the AR device; ascertaining, by the AR device, a current location or geographical area of the AR device based on the sensor data as received; (“In the exemplary embodiment depicted in the figure, there are shown, further to the components of Fig. 1A, a construction site self-positioning user interface (UI) and a GPS unit of the mobile device. The UI may allow of the user to provide his location within a given construction site by positioning himself over a 2D blueprint(s) of the construction site... The user selection may, for example, be obtained through a screen touch of the user at the relevant position on the blueprint presented on a touchscreen display of the device... User selected and/or mobile device GPS based positioning data may then be relayed to the Self-Localization Unit of the system server for allowing for, or assisting with, the positioning of the user and thus the mobile device within the actual construction site.”; Rozenberg, [0073], Fig. 2)
Fig. 2 illustrates that the mobile device includes a construction site self-positioning UI that allows the user to provide his location through the user’s touch input (receiving, by the AR device, sensor data) at the relevant position on a blueprint (indicative of a location of the AR device) displayed on the touchscreen display of the mobile device. The mobile device also includes a GPS unit that detects the user’s position (ascertaining, by the AR device, a current location or geographic area of the AR device based on the sensor data as received).
(“One or more sensors, including a camera, of a computerized device may be utilized to digitize a scene of a construction site within which a user of the device is present. Digitized scene data and/or features and feature sets in the digitized scene may be extracted... Extracted features and/or construction associated objects derived therefrom, identified within one or more of the 3-dimensional construction site model may indicate the specific construction site within which the computerized device user is present, the location of the computerized device user within the site and the orientation of the computerized device at that location.”; Rozenberg, [0060])
Also, one or more sensors, including a camera, produces a digitized scene of a construction site within which a user is present. Extracted features of the digitized scene data are identified within the 3D construction site model, which indicates the specific construction site within which the device of the user is present, the location of the device within the site and the orientation of the device at that location. Thus, digitized scene from the camera (sensor data) is used to extract features which are used indicate and ascertain the location of the device.
retrieving, by the AR device, geospatial information associated with both the current location or geographical area of the AR device and one or more of the construction components; (“The shown mobile device further includes Device Follow-up Sensors... using a combination of accelerometers and gyroscopes and/or magnetometers... Device Follow-up Sensors are utilized to follow the movement of the mobile space, deducting its ever changing position, orientation and/or viewing angle, as the user moves through the construction site...”; Rozenberg, [0075], Fig. 2)
Fig. 2 illustrates device follow-up sensors that use a combination of accelerometers, gyroscopes and/or magnetometers to follow determine the changing position of the mobile device as the user moves through the construction site (retrieving, by the AR device, geospatial information associated with both the current location or geographical area of the AR device).
(“In the figure, there is shown a scene digitizer installed onto or integrated into a mobile computerized device. The scene digitizer receives the output signals from a selection of shown mobile device sensors, including: a camera, a depth sensor, accelerometers, gyroscopes and/or magnetometers. Digitized scene data, for example in the form of a digital image and a depth map, of the viewed scene, is generated by the scene digitizer. Generated data is relayed to the system server and to the shown feature detector for search of potential construction objects related features. Potential features and their position within the digitized scene are relayed to the shown feature extractor for extraction of properties such as their: dimension, orientation, texture, color and/or the like.”; Rozenberg, [0077], Fig. 3A)
Fig. 3A illustrates that the mobile device includes a scene digitizer that receives signals from mobile device sensors such as, a camera, a depth sensor, accelerometers, gyroscopes and/or magnetometers. Scene data, generated by the scene digitizer, is relayed to for example, the feature detector for search of potential construction objects (construction components) related features. These potential features of the construction objects and their position within the scene (retrieving, by the virtual reality device or the augmented reality device, geospatial information associated with both the current location or geographical area of the AR device and one or more of the construction components) are relayed to the feature extractor, which extracts properties such as the object’s dimension, orientation, texture and color.
rendering, by the AR device, a second enhanced construction image that comprises additional content incorporating the retrieved geospatial information; (“The system server in the shown embodiment further includes a Server Rendering Engine for generating visual presentation rendering instructions for the augmentation of differences/deltas between the 3D model and the actual viewed/sensed construction site scene. The shown Rendering Instructions Relay Unit communicates the already generated rendering instructions to a Model Presentation Unit of the mobile device.”; Rozenberg, [0074])
Fig. 2 illustrates that the server includes a server rendering engine for generating rendering instructions for the augmentation of differences/deltas between the 3D model and the actual viewed construction site scene (second enhanced construction image that comprises additional content incorporating the retrieved geospatial information). The server also includes a rendering instruction relay unit that communicates the rendering instruction to the model presentation unit of the mobile device for display (rendering, by the virtual reality device or the augmented reality device, a second enhanced construction image that comprises additional content incorporating the retrieved geospatial information).
(“Object differences/errors/deltas may optionally be presented as a real-time visual overlay on the scene being displayed to the system user, optionally over the display of the mobile computerized device. Indicated object differences/errors/deltas may include, visually marking:... differences in the size, shape, position and/or orientation of objects or object-features identified as similar in both the scene and the model...”; Rozenberg, [0091], Figs. 1A and 7A)
For example, the object differences/error/deltas (additional information) may be presented as a real-time visual overlay on the scene being displayed to the user of the mobile device.
providing, by the AR device to a machine learning classification routine, feature classifiers containing characteristics for identifying pixels which represent the construction components within a panoramic view; (“The feature detector may analyze the acquired digital image to recognize potential features (e.g., construction related features which are part of a construction object – for example, 4 corners of a window) within it.”; Rozenberg, [0076], Fig. 1)
The feature detector (feature classifiers) of the mobile device analyzes the image and recognizes construction features which are part of a construction object, such as 4 corners of a window (characteristics... which represent the construction components).
(“Digitized scene data, for example in the form of a digital image and a depth map, of the viewed scene, is generated by the scene digitizer. Generated data is relayed to the system server and to the shown feature detector for search of potential construction objects related features. Potential features and their position within the digitized scene are relayed to the shown feature extractor for extraction of properties such as their: dimension, orientation, texture, color and/or the like. Extracted feature data is then relayed to the system server.”; Rozenberg, [0077], Fig. 3A)
Fig. 3A illustrates a mobile device which includes a scene digitizer including a feature detector and a feature extractor. The feature detector of the mobile device determines features of construction objects in the scene. These features are sent to the feature extractor for extraction of properties of the features, such as dimension, orientation, texture and color. The extracted feature data is then sent (providing... feature classifiers containing characteristics... which represent construction components) to the server.
(“A vector model processor... may include a visual object generator, for identifying objects in the digitized scene based on the detected and extracted features... In the figure, there is shown a vector model processor installed onto or integrated into a system server. The vector model processor receives digitized scene data and scene features’ parameters.”; Rozenberg, [0079], Fig. 4A)
Fig. 4A illustrates a server, which includes a vector model processor including a visual object generator and an object identification processor. The vector model processor (which includes the object identification engine), of the server, receives the extracted features, from the mobile device and uses these features to identify objects in the digitized scene.
(“... An object identification engine for identifying partially-completed construction objects from one or more of the digital images of the real-world construction scene, wherein the identification is performed on two levels/stages: (a) Identifying the current construction stage... the engine identifies the unique features; construction associated features may, for example, be derived by a proprietary, or a third party... machine learning engine, ”; Rozenberg, [0099], [0100], Fig. 4A)
The object identification engine, of the server, includes machine learning. Thus, the extracted feature data that is received by the vector model processor, which includes the object identification engine, also includes machine learning (providing... to a machine learning classification routine, feature classifiers containing characteristics... that represent construction components. Rozenberg is silent regarding pixels, however, Connary teaches this limitation.
(“Machine learning-based object identification, segmentation, and/or labeling algorithms can be used to identify the 2D/3D boundaries, geometry, type and health of objects, components, and scene or locations of interest... Deep Convolutional Neural Networks (DCNNs) can be used to assigning a label to one or more portions of an image (e.g., bounding box, region enclosed by a contour, or a set of pixels creating a regular or irregular shape) that include a given object, feature, scene, or location of interest, collection of the physical assets of interest, or components or features on or relevant to the asset(s).”; Connary, col. 52, lines 49-65)
The machine learning based object identification, is used to assign labels to (identify), for example, a set of pixels of a given object. Connary is combined with Rozenberg such that the machine learning of Rozenberg includes the machine learning of Connary that identifies, for example, a set of pixels of representing an object or collection of the physical assets of interest, where the object and the collection of physical assets of interest are considered to be construction components. Rozenberg is silent regarding a panoramic view, however, Trehan teaches this limitation. Trehan teaches this limitation.
(“... the mixed reality along with the artificial intelligence is used in image processing where the dimensions of the objects in the captured panoramic image are selected automatically, which are used for prediction... Referring to Fig. 1, the camera present in the smartphone can be used to capture the panoramic image of the house of the user, where the objects which has to be removed or replaced are located.”; Trehan, p. 6, 1st-2nd para under “Detailed description of the Invention”)
Fig. 1 illustrates that the smartphone has captured a panoramic image of the house of the user, which include objects to be removed or replaced.
(“The present invention uses the deep learning technique with which the user can train the objects’ dimensions to the image clicked in panoramic view. The main idea behind the invention is that the objects having sharp edges get selected automatically. With the help of the present invention, the user can use virtual reality feature to shrink or adjust the dimensions of matching object’s image on the website to the panoramic image and finally replace and remove the objects in the processed image.”; Trehan, p. 7, 3rd para, Fig. 2)
Deep learning is used in combination with virtual reality to select objects (superfluous portions of the image) in the panoramic image (panoramic view), compare these objects to matching objects (sample pixels) on the website and remove (removing superfluous portions of an image) and/or replace these objects in the image. Therefore, it would have been obvious to one of ordinary skill in the art at the time of the effective filing date, to modify the method of Rozenberg by adding the feature of providing, by the virtual reality device or the augmented reality device to a machine learning classification routine, feature classifiers containing characteristics for identifying pixels which represent the construction components, in order to identify the 2D/3D boundaries, geometry, type and health of objects, components and scene or locations of interest so that it can be replaced by an object representing a physical asset or elements thereof with corresponding semantic data, as taught by Connary (col. 52, lines 49-58) and in order to remove or replace selected objects to obtain a new processed image, as taught by Trehan (p. 8, 4th-5th paras).
Rozenberg and Connary are silent regarding using the feature classifiers, identifying, by the AR device, construction features within the panoramic view and removing superfluous portions of an image unrelated to the identified construction features, wherein removing the superfluous portions of the image is performed using at least one of a spatial heuristic algorithm and a machine learning technique that compare pixels in a panoramic view to sample pixels, however Weng and Trehan teach
and using the feature classifiers, identifying, by the AR device, construction features within the panoramic view and removing superfluous portions of an image unrelated to the identified construction features, (“Obtain an RGB image of the target area, and use the R, G, and B values of the RGB image as the input of the color classification model of the neural network, and input the color classification model of the neural network. The pixels of the RGB image are classified, and the background pixels are removed to obtain an image of the target pixel area. The process of removing background pixels includes: reading the RGB values of the target pixel points and inputting them to the three input channels of the neural network, and computing the neural network to obtain whether the pixel point is a target pixel point or a non-target pixel point. Then remove the non-target pixels in the picture, that is, remove the background pixels.”; Weng, p. 14, last para)
The color classification model of the neural network (feature classifiers) classifies pixels of the RGB image, which includes the target area. Based on this classification (using the feature classifiers), the target area pixels are identified (identified construction features) and distinguished from the background pixels (superfluous portions of an image), and then the background pixels are removed (removing superfluous portions of and image unrelated to the identified construction features) to obtain an image of the target pixel area. Wang is silent regarding a panoramic view, however, Trehan teaches
(“... the mixed reality along with the artificial intelligence is used in image processing where the dimensions of the objects in the captured panoramic image are selected automatically, which are used for prediction... Referring to Fig. 1, the camera present in the smartphone can be used to capture the panoramic image of the house of the user, where the objects which has to be removed or replaced are located.”; Trehan, p. 6, 1st-2nd para under “Detailed description of the Invention”)
Fig. 1 illustrates that the smartphone has captured a panoramic image of the house of the user, which include objects to be removed or replaced.
(“The present invention uses the deep learning technique with which the user can train the objects’ dimensions to the image clicked in panoramic view. The main idea behind the invention is that the objects having sharp edges get selected automatically. With the help of the present invention, the user can use virtual reality feature to shrink or adjust the dimensions of matching object’s image on the website to the panoramic image and finally replace and remove the objects in the processed image.”; Trehan, p. 7, 3rd para, Fig. 2)
Deep learning is used in combination with virtual reality to select objects (superfluous portions of the image) in the panoramic image (panoramic view), compare these objects to matching objects (sample pixels) on the website and remove (removing superfluous portions of an image) and/or replace these objects in the image.
Rozenberg and Connary are silent regarding removing the superfluous portions of the image is performed using at least one of a spatial heuristic algorithm and a machine learning technique that compare pixels in a panoramic view to sample pixels, however, Weng and Trehan teach
wherein removing the superfluous portions of the image is performed using at least one of a spatial heuristic algorithm applied to the panoramic view and a machine learning technique that compares pixels in the panoramic view to sample pixels. (“Obtain an RGB image of the target area, and use the R, G, and B values of the RGB image as the input of the color classification model of the neural network, and input the color classification model of the neural network. The pixels of the RGB image are classified, and the background pixels are removed to obtain an image of the target pixel area. The process of removing background pixels includes: reading the RGB values of the target pixel points and inputting them to the three input channels of the neural network, and computing the neural network to obtain whether the pixel point is a target pixel point or a non-target pixel point. Then remove the non-target pixels in the picture, that is, remove the background pixels.”; Weng, p. 14, last para)
The color classification model of the neural network (machine learning technique) classifies pixels of the RGB image, which includes the target area. Based on this classification, the target area pixels are identified and distinguished (compares pixels to sample pixels) from the background pixels (superfluous portions of an image), and then the background pixels are removed (removing superfluous portions of the image) to obtain an image of the target pixel area. Wang is silent regarding a panoramic view, however, Trehan teaches
(“... the mixed reality along with the artificial intelligence is used in image processing where the dimensions of the objects in the captured panoramic image are selected automatically, which are used for prediction... Referring to Fig. 1, the camera present in the smartphone can be used to capture the panoramic image of the house of the user, where the objects which has to be removed or replaced are located.”; Trehan, p. 6, 1st-2nd para under “Detailed description of the Invention”)
Fig. 1 illustrates that the smartphone has captured a panoramic image of the house of the user, which include objects to be removed or replaced.
(“The present invention uses the deep learning technique with which the user can train the objects’ dimensions to the image clicked in panoramic view. The main idea behind the invention is that the objects having sharp edges get selected automatically. With the help of the present invention, the user can use virtual reality feature to shrink or adjust the dimensions of matching object’s image on the website to the panoramic image and finally replace and remove the objects in the processed image.”; Trehan, p. 7, 3rd para, Fig. 2)
Deep learning is used in combination with virtual reality to select objects (superfluous portions of the image) in the panoramic image (panoramic view), compare these objects to matching objects (sample pixels) on the website (compares pixels in the panoramic view to sample pixels) and remove (removing superfluous portions of the image) and/or replace these objects in the image. Therefore, it would have been obvious to one of ordinary skill in the art at the time of the effective filing date, to modify the method of Rozenberg by adding the feature of using the feature classifiers, identifying, by the AR device, construction features within the panoramic view and removing superfluous portions of an image unrelated to the identified construction features, wherein removing the superfluous portions of the image is performed using at least one of a spatial heuristic algorithm and a machine learning technique that compare pixels in a panoramic view to sample pixels, in order to continuously learn the objects in the image and establish a neural network color classification model to realize the recognition of the objects in the image, as taught by Weng (p. 15, 2nd para), and in order to remove or replace selected objects to obtain a new processed image, as taught by Trehan (p. 8, 4th-5th paras)
Claim 36 is a system analogous to the method of claim 26, is similar in scope and is rejected under the same rationale. Re: claim 36, Rozenberg teaches
36. (Currently Amended) A system for generating a graphical user interface (GUI) for facilitating compliance with building or workplace health and safety standards, comprising: a computer processing system comprising a hardware processor and memory having program instructions stored thereon that, when executed, direct the computing processing system to
perform the method of claim 26 (“... there may be provided an exemplary system and method for interactive visualization of construction site plans... The exemplary system... may comprise... a processor and a memory for storing instructions executable by the processor for display virtual objects.”; Rozenberg, [0114], [0115])
The system comprises a processor (hardware processor) and memory for storing instructions (having program instructions stored thereon) executable by the processor (when executed, direct the computing processing system to).
Re: claim 27, Rozenberg, Connary, Weng and Trehan teach
27. (Previously Presented) The method of claim 26, wherein the schematic representation of the construction information comprises at least one of: an image, a visual mark, a dimension indicator, a risk indicator, an animation, and a color indicator. (“The UI may allow for the user to provide his location with a given construction site by positioning himself over a 2D blueprint(s) of the construction site, retrieved by the system server from the 3D construction site models database and relayed to the mobile device.”; Rozenberg, [0073], Fig. 2)
The mobile device displays blueprints (an image) of the construction site (the schematic representation of the construction information comprises: at least one of an image, a visual mark, a dimension indicator, a risk indicator, an animation, and a color indicator).
Re: claim 28, Rozenberg, Connary, Weng and Trehan teach
28. (Previously Presented) The method of claim 26, further comprising: detecting, by the virtual reality device or the augmented reality device, a change in location of the virtual reality device or the augmented reality device from the current location to a new location associated with new geospatial information, wherein detecting the change in location comprises ascertaining a change from the geospatial information to the new geospatial information. (“The shown mobile device further includes Device follow-up Sensors... using a combination of accelerometers and gyroscopes and/or magnetometers. Device Follow-up Sensors are utilized to follow the movement of the mobile space, deducting its ever changing position, orientation and/or viewing angle, as the user moves through the construction site...”; Rozenberg, [0075], Fig. 2)
Fig. 2 illustrates that the mobile device includes device follow-up sensors that detect the ever changing position of the mobile device (detecting, by the virtual reality or the augmented reality device, a change in location of the virtual reality device or the augmented reality device) as the user moves through the construction site (from the current location to a new location associated with new geospatial information, wherein detecting the change in location comprises ascertaining a change from the geospatial information to the new geospatial information).
Claim(s) 37 is/are rejected under 35 U.S.C. 103 as being unpatentable over Rozenberg, Connary, Weng and Trehan as applied to claim 36 above, and further in view of Bellaish et al. U.S. Pub. No. 2020/0151833.
Re: claim 37, Rozenberg, Connary, Weng and Trehan are silent regarding the annotations on the image comprise dimensions from one element to another element, in accordance with a relevant standard, wherein the dimensions are presented as a graphical representation superimposed on the image of the building, however, Bellaish teaches
37. (Previously Presented) The system of claim 36, wherein the annotations on the image comprise dimensions from one element to another element, in accordance with a relevant standard, wherein the dimensions are presented as a graphical representation superimposed on the image of the building. (“For example, an overlay presenting desired dimensions of an object (such as a room, a wall, a doorway, a window, a tile, an electrical box, etc.) may be presented over a depiction of the object, for example as textual information specifying the desired dimensions and/or the actual dimensions, as a line or a shape demonstrating the desired dimensions, and so forth.”; Bellaish, [0147])
An overlay, that includes the desired dimensions of an object (annotations on the image comprise dimensions from one element to another element, in accordance with a relevant standard), is presented over the image of the object (dimensions are presented as a graphical representation superimposed on the image of the building). Therefore, it would have been obvious to one of ordinary skill in the art at the time of the effective filing date, to modify the method of Rozenberg by adding the feature of the annotations on the image comprise dimensions from one element to another element, in accordance with a relevant standard, wherein the dimensions are presented as a graphical representation superimposed on the image of the building, in order to present identified discrepancies of, for example, dimensions of objects in image data , as taught by Bellaish ([0147]).
Response to Arguments
Applicant's arguments filed 2/23/2026 have been fully considered but they are not persuasive. Applicant argues:
“However, Rozenberg does not disclose retrieving geospatial information associated with both the current location or geographical area of the augmented reality device and one or more of the construction components, as expressly required by claim 26. The positioning functionality of Rozenberg appears to rely on user selection within a blueprint and optionally GPS assisted localization to determine device pose within a predefined construction site model. The extracted scene features and positional data are used to align a live scene with a stored model and to identify deltas between corresponding objects. This process determines relative position and orientation within a construction model, but it does not involve retrieving geospatial information associated with a geographical area and associating such geospatial information with specific construction components.”
Examiner disagrees. Rozenberg illustrates in Fig. 2 that the device follow-up sensors use a combination of accelerometers, gyroscopes and/or magnetometers to follow determine the changing position of the mobile device as the user moves through the construction site (retrieving, by the AR device, geospatial information associated with both the current location or geographical area of the AR device). (Rozenberg, [0075], Fig. 2). Fig. 3A illustrates that the mobile device (AR device) includes a scene digitizer that receives signals from mobile device sensors such as, a camera, a depth sensor, accelerometers, gyroscopes and/or magnetometers. Scene data, generated by the scene digitizer, is relayed to for example, the feature detector for search of potential construction objects (construction components) related features. These potential features of the construction objects and their position within the scene (retrieving, by AR device, geospatial information associated with both the current location or geographical area of the AR device and one or more of the construction components) are relayed to the feature extractor, which extracts properties such as the object’s dimension, orientation, texture and color. (Rozenberg, [0077], Fig. 3A).
Applicant's arguments filed 2/23/2026 have been fully considered but they are not persuasive. Applicant argues:
“Claim 26 requires retrieval of geospatial information that is associated with both of the ascertained geographical area of the device and with one or more construction components, followed by rendering a second enhanced construction image incorporating that geospatial information. Rozenberg does not show or suggest retrieving environmental, regulatory, climate based, or other geographically dependent data sets that are associated with specific construction components and incorporated into an enhanced image. Determining pose within a model, as described by Rozenberg, is fundamentally different from retrieving geospatial information associated with a geographical area and tying that information to particular construction components.”
Examiner disagrees. In response to applicant's argument that the references fail to show certain features of the invention, it is noted that the features upon which applicant relies (i.e., retrieving environmental, regulatory, climate based, or other geographically dependent data sets that are associated with specific construction components and incorporated into an enhanced image) are not recited in the rejected claim(s). Although the claims are interpreted in light of the specification, limitations from the specification are not read into the claims. See In re Van Geuns, 988 F.2d 1181, 26 USPQ2d 1057 (Fed. Cir. 1993).
Applicant's arguments filed 2/23/2026 have been fully considered but they are not persuasive. Applicant argues:
“Moreover, Rozenberg also fails to disclose the claimed image processing involving removal of superfluous portions of a panoramic view using either a spatial heuristic algorithm applied to the panoramic view or a machine learning technique that compares pixels in the panoramic view to sample pixels. Rozenberg describes digitizing a scene and extracting features for purposes of object identification and model comparison. Feature detection in Rozenberg is used to recognize construction related features and to compare them against a stored model. Rozenberg does not disclose, however, classifying pixels within a panoramic image, comparing panoramic pixels to sample pixels, or removing background or superfluous portions of a panoramic view. The overlay described in Rozenberg adds information to a scene but does not remove portions of the image. Feature extraction for alignment is not equivalent to pixel level segmentation and elimination of non-relevant image regions using spatial heuristics or machine learning comparison against sample pixels. Because these limitations are absent from Rozenberg, amended claim 26 is not anticipated. Dependent claims 27, 28, 36, and 37 incorporate these additional limitations and are likewise not anticipated.”
Examiner disagrees. Weng teaches this limitation. Weng and Trehan teach this limitation. Weng teaches that the color classification model of the neural network (machine learning technique) classifies pixels of the RGB image, which includes the target area. Based on this classification, the target area pixels are identified and distinguished (compare pixels to sample pixels) from the background pixels (superfluous portions of an image), and then the background pixels are removed (removing superfluous portions of the image) to obtain an image of the target pixel area. (Weng, p. 14, last para). Trehan illustrates in Fig. 1 that the smartphone has captured a panoramic image of the house of the user, which include objects to be removed or replaced. (Trehan, p. 6, 1st-2nd para under “Detailed description of the Invention”). Deep learning is used in combination with virtual reality to select objects (superfluous portions of the image) in the panoramic image (panoramic view), compare these objects to matching objects (sample pixels) on the website (compare pixels in a panoramic view to sample pixels) and remove (removing superfluous portions of the image) and/or replace these objects in the image. (Trehan, p. 7, 3rd para, Fig. 2). In response to applicant's argument that the references fail to show certain features of the invention, it is noted that the features upon which applicant relies (i.e., classifying pixels within a panoramic image, comparing panoramic pixels to sample pixels, or removing background or superfluous portions of a panoramic view) are not recited in the rejected claim(s). Although the claims are interpreted in light of the specification, limitations from the specification are not read into the claims. See In re Van Geuns, 988 F.2d 1181, 26 USPQ2d 1057 (Fed. Cir. 1993).
Applicant's arguments filed 2/23/2026 have been fully considered but they are not persuasive. Applicant argues:
“Even if the cited references are to properly combinable, which Applicant does not concede, they do not render amended claim 26 obvious. The cited references address different technical problems and operate in materially different contexts. For instance, Rozenberg is directed to aligning a live construction scene with a stored model and visually indicating differences between the two. Connary discusses digital twin implementations and machine learning based object identification in a broader asset monitoring context. Weng describes color based neural network classification of pixels for background removal. Trehan relates to panoramic image capture and replacement of selected objects in a residential or consumer environment. Price generally lists types of machine learning models.”
Examiner disagrees. In response to applicant's argument that Rozenberg, Connary, Weng and Trehan are nonanalogous art, it has been held that a prior art reference must either be in the field of the inventor’s endeavor or, if not, then be reasonably pertinent to the particular problem with which the inventor was concerned, in order to be relied upon as a basis for rejection of the claimed invention. See In re Oetiker, 977 F.2d 1443, 24 USPQ2d 1443 (Fed. Cir. 1992). In this case, the cited references are all analogous art. Rozenberg is being used to teach, for example a mobile computing device (AR device) that displays construction images, construction components, etc., geospatial information and machine learning classification. Connary is being used to teach, for example, that machine learning object identification is used to assign labels (identify) to a set of pixels of a given object. Weng is being used to teach, for example, using a neural network to classify pixels in the RGB image, which includes the target area, where the target area is distinguished from the background pixels and the background area is removed (removing superfluous portions of the image). Trehan is being used to teach, for example, a panoramic image which includes using deep learning to remove objects (removing superfluous portions of a panoramic image). For more details regarding these references, please see the rejection for claim 26
Applicant's arguments filed 2/23/2026 have been fully considered but they are not persuasive. Applicant argues:
“The Office's rejection extracts isolated teachings from these disparate systems and combines them to reconstruct the claimed invention. However, the Office Action does not provide a specific articulated reason why a person of ordinary skill in the art would modify the construction alignment and delta overlay system of Rozenberg to incorporate pixel level background removal from a panoramic view using spatial heuristics or comparison of panoramic pixels to sample pixels, nor why such modifications would be made in conjunction with retrieving geospatial information associated both with the device's geographical area and with particular construction components. The references do not suggest integrating these features into a unified augmented reality pipeline as claimed, and these features are not combinable without undue experimentation.”
Examiner disagrees. In response to applicant’s argument that there is no teaching, suggestion, or motivation to combine the references, the examiner recognizes that obviousness may be established by combining or modifying the teachings of the prior art to produce the claimed invention where there is some teaching, suggestion, or motivation to do so found either in the references themselves or in the knowledge generally available to one of ordinary skill in the art. See In re Fine, 837 F.2d 1071, 5 USPQ2d 1596 (Fed. Cir. 1988), In re Jones, 958 F.2d 347, 21 USPQ2d 1941 (Fed. Cir. 1992), and KSR International Co. v. Teleflex, Inc., 550 U.S. 398, 82 USPQ2d 1385 (2007). In this case, Weng and Trehan are combined with Rozenberg in order to continuously learn the objects in the image and establish a neural network color classification model to realize the recognition of the objects in the image, as taught by Weng (p. 15, 2nd para), and in order to remove or replace selected objects to obtain a new processed image, as taught by Trehan (p. 8, 4th-5th paras).
Applicant's arguments filed 2/23/2026 have been fully considered but they are not persuasive. Applicant argues:
“The system of Rozenberg operates by detecting features and comparing them to a model in order to highlight differences. The color classification model of Weng removes background pixels in a generic RGB image context. Trehan's panoramic image processing relates to replacing selected objects in a captured panoramic view for visualization purposes. There is no teaching or suggestion in these references that the pixel classification and removal techniques should be incorporated into Rozenberg's augmented reality construction system for the purpose of identifying construction components within a panoramic view and removing superfluous portions unrelated to those identified construction features. Nor is there a teaching that such image processing should be combined with retrieval of geospatial information tied both to the device's ascertained geographical area and to specific construction components and then rendered as a second enhanced construction image. The rejection therefore relies on hindsight reconstruction of Applicant's claimed method by assembling elements from unrelated references after reviewing the claim. Under KSR, an obviousness rejection must provide an articulated reasoning with rational underpinning to support the proposed combination. Here, the Office does not explain why the proposed modifications would have been made, how they would have been integrated into Rozenberg's architecture, or why a person of ordinary skill would have expected such a combination to yield the claimed result. Accordingly, amended claim 26, and dependent claims 27, 28, 36, and 37, are not rendered obvious by the cited combination.”
Examiner disagrees. In response to applicant’s argument that there is no teaching, suggestion, or motivation to combine the references, the examiner recognizes that obviousness may be established by combining or modifying the teachings of the prior art to produce the claimed invention where there is some teaching, suggestion, or motivation to do so found either in the references themselves or in the knowledge generally available to one of ordinary skill in the art. See In re Fine, 837 F.2d 1071, 5 USPQ2d 1596 (Fed. Cir. 1988), In re Jones, 958 F.2d 347, 21 USPQ2d 1941 (Fed. Cir. 1992), and KSR International Co. v. Teleflex, Inc., 550 U.S. 398, 82 USPQ2d 1385 (2007). In this case, Weng and Trehan are combined with Rozenberg in order to continuously learn the objects in the image and establish a neural network color classification model to realize the recognition of the objects in the image, as taught by Weng (p. 15, 2nd para), and in order to remove or replace selected objects to obtain a new processed image, as taught by Trehan (p. 8, 4th-5th paras). In response to applicant's argument that the examiner's conclusion of obviousness is based upon improper hindsight reasoning, it must be recognized that any judgment on obviousness is in a sense necessarily a reconstruction based upon hindsight reasoning. But so long as it takes into account only knowledge which was within the level of ordinary skill at the time the claimed invention was made, and does not include knowledge gleaned only from the applicant's disclosure, such a reconstruction is proper. See In re McLaughlin, 443 F.2d 1392, 170 USPQ 209 (CCPA 1971).
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DONNA J RICKS whose telephone number is (571)270-7532. The examiner can normally be reached on M-F 7:30am-5pm EST (alternate Fridays off).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Devona Faulk can be reached on 571-272-7776. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see https://ppair-my.uspto.gov/pair/PrivatePair. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Donna J. Ricks/Examiner, Art Unit 2612
/DEVONA E FAULK/Supervisory Patent Examiner, Art Unit 2618
Prosecution Insights