DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged that application claims priority to foreign application with application number JP2023-169997 dated 09/29/2023. Copies of certified papers required by 37 CFR 1.55 have been received. Priority is acknowledged under 35 USC 119(e) and 37 CFR 1.78.
Drawings
The drawings are objected to as failing to comply with 37 CFR 1.84(p)(5) because they include the following reference character(s) not mentioned in the description:
“24” from Figure 1
“Bp”, “Sa”, and Gi” from Figure 6
All features from Figures 7 and 8
“Ob1” and “Ob2” from Figure 9
“4” from Figure 10
The drawings are objected to because: “Wp” in Figure 1 should read “Wp1” (see para 72).
Corrected drawing sheets in compliance with 37 CFR 1.121(d), or amendment to the specification to add the reference character(s) in the description in compliance with 37 CFR 1.121(b) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Claim Objections
Claims 1, 7, and 8 is objected to because of the following informalities: “position information of an imaging position and the imaging posture of the new image” should read “position information of the imaging position and the imaging posture of the new image”. Appropriate correction is required.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are:
“storage unit” in claim 1
“imaging unit” in claims 1, 2
“posture detection unit” in claim 1
“feature extraction unit” in claim 1
“feature matching unit” in claim 1
“imaging position calculation unit” in claim 1
“selection unit” in claims 1, 4, 5
“guide calculation unit” in claims 1, 6
“display unit” in claims 2, 4
“display control unit” in claims 2-4
“imaging range calculation unit” in claim 2
“voice unit” in claim 6
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 2-4 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 2 recites the limitation “a new image”, which renders the claims indefinite. It is unclear whether the new image displayed on the display unit is the same “new image” introduced in claim 1 or a different, newly captured image. The metes and bounds of what is considered the “a new image” in claim 2 is not clearly defined by the claim or the specification, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention. For examination purposes, references to “the new image” will be interpreted to be the new image introduced in claim 1 and references to the displayed “a new image” will be interpreted to be any image captured by the system. Dependent claims 3 and 4 are similarly rejected due to their dependence on a rejected base claim.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 5, 7, and 8 are rejected under 35 U.S.C. 103 as being unpatentable over Kahle et al. (U.S. Patent No. 11,512,956 B2 – cited in IDS 04/09/2025), hereinafter Kahle, in view of Finn et al. (U.S. Patent No. 2014/0210856 A1), hereinafter Finn.
Regarding claim 1, Kahle teaches a guide system to assist a worker with work in a work site (Kahle, col 8, ln 41-46: “By showing the user a virtual object in relation to the environment, construction tasks can be simplified. In the example above, a construction worker can see, through the optical display 120, a virtual object (e.g., a dot) where an anchor point is to be located on a physical wall”), comprising:
a storage unit (Kahle, col 25, ln 55: “Storage subsystem 2004”) for storing design information including position information of scheduled installation positions of a plurality of objects or points, which are elements constituting a building to be constructed in the work site (Kahle, positions of where the virtual objects should be located, col 22, ln 62-65: “The optical display 120 produces a virtual object 1516 to appear in relation to the environment. The virtual object 1516 in FIG. 16 is an outline for an electrical box of a light switch positioned near the entryway”), and a plurality of existing feature points and position information of a site reference object to be used as a reference of a position in the work site acquired as a result of a survey by a surveying apparatus (Kahle, targets on the wall in FIG. 15, col 20, ln 32: “using a target 1504 on a surface as truth for measurements”; determined by base station, which performs surveying, col 20, ln 48-50: “measure a relative location of the target 1504 to the base station 108 (e.g., using the base station 108)”); and
a guide terminal held by the worker having an imaging unit for capturing a new image (Kahle, col 20, ln 34-36: “The wearable device 904 comprises a camera and an optical display (e.g., a camera and an optical display as part of an augmented-reality system)”), and a posture detection unit for detecting imaging direction information of the new image (Kahle, position of device in the environment indicates direction of the image, col 11, ln 58-60: “The camera 704 acquires image data (e.g., pictures) used to determine a position of the wearable device 104 within the environment”), the guide terminal including:
a feature extraction unit extracting a plurality of feature points from the new image (Kahle, extraction of targets 1504 in FIG. 15, see next citation);
a feature matching unit using the plurality of feature points included in the new image to extract a predetermined number or more common feature points, which are in common with the feature points, from among the plurality of existing feature points stored on the storage unit for association (Kahle, orient the local map, including targets, to the location of targets in relation to the base, col 20, ln 50-58: “generate a local map based on a plurality of images acquired by the camera (e.g., of the wearable device 904), wherein the local map includes a relative location of the target 1504 to the wearable device 904; and/or orient the local map to an environment of the base station 108 based on the relative location of the target 1504 to the base station 108, the relative location of the target 1504 to the wearable device 904, and/or a relative location of the base station 108 to the environment”);
an imaging position calculation unit using position information of the common feature points, which are associated, and the new image, to calculate position information (Kahle, targets are used to orient the local map’s position in the environment, col 21, ln 5-14: “a feature in the environment could be used as a target…The wearable device 904 measures relative position of the wearable device 904 in relation to the feature (e.g., corner 1512), and can orient the local map in relation to the environment based on the relation of the wearable device 904 to the feature and the relation of the feature to the base station 108”; targets are used to determine device orientation, which includes the camera, see col 22, ln 58-61);
a selection unit for selecting a work target object or a work target point that should be guided as to its scheduled installation position from among the plurality of objects or points stored on the storage unit (Kahle, selection of a virtual object 1516 to be displayed, see next citation); and
a guide calculation unit for outputting, to the worker, based on the position information of the scheduled installation position of the work target object or the work target point, which is selected, and the position information of an imaging position and the imaging posture of the new image, guide information from the imaging position of the new image to the scheduled installation position of the work target object or the work target point (Kahle, 1516 in FIG. 15, col 10, ln 45-51: “Points 320 are virtual objects generated by the optical display. The points 320 are generated by the optical display and presented to a user of the wearable device 104 to appear in relation to the environment 316. Accordingly, the points 320 appear to the user to be on a wall in the environment 316, but another person standing next to the user of the wearable device 104 would not see the points 320”; providing a visual outline provides guide information, col 22, ln 63-66: “The virtual object 1516 in FIG. 16 is an outline for an electrical box of a light switch positioned near the entryway. The virtual object 1516 can be positioned in relation to features in the environment”; since the virtual object is in the image frame, the guide information is based on the image position information).
While Kahle teaches that the guide information is based on image position information (see last citation) and that the common feature points are used to determine the camera’s orientation in the environment (Kahle, col 22, ln 58-61: “Images of features acquired by a camera of the wearable device provide information to the wearable device about orientation (e.g., with or without further data acquired by an IMU”), Kahle fails to explicitly teach an imaging position calculation unit using position information of the common feature points, which are associated, and the new image, to calculate position information of an imaging position and an imaging posture of the new image (emphasis added).
However, Finn teaches a similar guide system (Finn, para 17: “As an illustration, to visualize internal elements (e.g., studs, pipes, or conduits) hidden behind a finished wall, a 3D digital model of internal elements can be generated. The 3D digital model can be generated using a blueprint of a building which is used to construct the building”). Finn teaches a captured image (Finn, para 33: “a camera 141, which can be used to detect and capture an image of the marker 103”; see FIG. 3E, para 41: “FIG. 3E illustrates a schematic diagram representing an augmented reality image shown on a display screen 345 of a mobile device”), position information of common feature points (Finn, marker, para 32: “marker 103 can be placed at precise, surveyed coordinates on the external element in the real environment using the surveying equipment 112. Thus, the marker placed on the external element 102 can be used as an anchor point for augmented reality visualization of the 3D digital model of the internal elements 101”), and an imaging position calculation unit using position information of the feature points, which are associated, and the new image, to calculate position information of an imaging position and an imaging posture of the new image (Finn, position and orientation of the displayed scene based on the marker orientation, para 81: “As the user walks around the project site or tilts the mobile device, the plane (i.e., physical surface) at which the physical marker 103 is affixed can change its orientation with the camera image plane. In an embodiment, when such a change is detected by the mobile device, a matrix transformation may be performed by the processor of the mobile device, and the scene seen through the mobile device can be adjusted to match the change in the mobile device position and orientation”).
Kahle discloses a base method for determining the position and orientation of images in the environment in order to display virtual objects in relation to the displayed image frame, but does not specify specific methods for using common feature points to do so. Finn teaches a known technique of utilizing markers (targets) to determine the position and posture of a captured image. A person having ordinary skill in the art, before the effective filing date of the claimed invention, could have applied the known technique, as taught by Finn, in the same way to the system taught by Kahle and achieved predictable results of using an imaged marker to determine the position and posture of a captured image. Doing so allows for the correct visualizing of virtual objects placed in the field of view of the image (Finn, para 81: “This way, the 3D digital model of the internal elements can still be visualized in the geometrically correct orientation with respect to the real scenes as a user moves around. By changing the viewing angle of the 3D digital model of the internal elements, the separation of the internal elements may be better visualized. For example, if the internal elements were pipes underground, then viewing the underground pipes at an angle using a mobile device may allow visualization of separation and stacking of the pipes compared to viewing them straight from the top”).
Regarding claim 5 (dependent on claim 1), Kahle in view of Finn teaches wherein the selection unit is configured to automatically select the work target object or the work target point, which should be selected, based on the position information of the imaging position of the new image (Kahle, Guide information is provided based on features in the environment, see col 22, ln 63-66, and are therefore selected based on the position of the image frame. For example, a target point out of view would not be selected for virtual object display if outside the position of the current image field of view.).
Regarding claim 7, Kahle teaches a guide method of assisting a worker with work in a work site (Kahle, col 8, ln 41-46: “By showing the user a virtual object in relation to the environment, construction tasks can be simplified. In the example above, a construction worker can see, through the optical display 120, a virtual object (e.g., a dot) where an anchor point is to be located on a physical wall”), comprising steps of:
storing, on a storage unit (Kahle, col 25, ln 55: “Storage subsystem 2004”), design information including a plurality of objects or points, which are elements constituting a building to be constructed in the work site, and position information of their scheduled installation positions (Kahle, positions of where the virtual objects should be located, col 22, ln 62-65: “The optical display 120 produces a virtual object 1516 to appear in relation to the environment. The virtual object 1516 in FIG. 16 is an outline for an electrical box of a light switch positioned near the entryway”), and a plurality of existing feature points and position information of a site reference object to be used as a reference of a position in the work site acquired as a result of a survey by a surveying apparatus (Kahle, targets on the wall in FIG. 15, col 20, ln 32: “using a target 1504 on a surface as truth for measurements”; determined by base station, which performs surveying, col 20, ln 48-50: “measure a relative location of the target 1504 to the base station 108 (e.g., using the base station 108)”);
capturing a new image with a guide terminal having an imaging unit used by the worker (Kahle, col 20, ln 34-36: “The wearable device 904 comprises a camera and an optical display (e.g., a camera and an optical display as part of an augmented-reality system)”; col 11, ln 58-60: “The camera 704 acquires image data (e.g., pictures) used to determine a position of the wearable device 104 within the environment”);
extracting a plurality of feature points from the new image with a feature extraction unit (Kahle, extraction of targets 1504 in FIG. 15, see next citation);
matching features, with a feature matching unit, by using the plurality of feature points included in the new image to extract a predetermined number or more common feature points, which are in common with the feature points included in the new image, from among the plurality of existing feature points stored on the storage unit for association (Kahle, orient the local map, including targets, to the location of targets in relation to the base, col 20, ln 50-58: “generate a local map based on a plurality of images acquired by the camera (e.g., of the wearable device 904), wherein the local map includes a relative location of the target 1504 to the wearable device 904; and/or orient the local map to an environment of the base station 108 based on the relative location of the target 1504 to the base station 108, the relative location of the target 1504 to the wearable device 904, and/or a relative location of the base station 108 to the environment”);
calculating, with an imaging position calculation unit, position information (Kahle, targets are used to orient the local map’s position in the environment, col 21, ln 5-14: “a feature in the environment could be used as a target…The wearable device 904 measures relative position of the wearable device 904 in relation to the feature (e.g., corner 1512), and can orient the local map in relation to the environment based on the relation of the wearable device 904 to the feature and the relation of the feature to the base station 108”; targets are used to determine device orientation, which includes the camera, see col 22, ln 58-61);
selecting, with a selection unit, a work target object or a work target point that should be guided as to its scheduled installation position from among the plurality of objects or points stored on the storage unit (Kahle, selection of a virtual object 1516 to be displayed, see next citation); and
guiding by outputting, with a guide calculation unit, to the worker, based on absolute position information of the scheduled installation position of the work target object or the work target point (Kahle, virtual object is presented based on its location in the environment, col 23, ln 45-49: “A virtual object is presented (e.g., to a user) on a display of the wearable device in relation to the environment, and/or one or more coordinates of a physical object is measured, based on orienting the local map of the wearable device with the environment”), which is selected, and the position information of an imaging position and the imaging posture of the new image, guide information from the imaging position of the new image to the scheduled installation position of the work target object or the work target point (Kahle, 1516 in FIG. 15, col 10, ln 45-51: “Points 320 are virtual objects generated by the optical display. The points 320 are generated by the optical display and presented to a user of the wearable device 104 to appear in relation to the environment 316. Accordingly, the points 320 appear to the user to be on a wall in the environment 316, but another person standing next to the user of the wearable device 104 would not see the points 320”; providing a visual outline provides guide information, col 22, ln 63-66: “The virtual object 1516 in FIG. 16 is an outline for an electrical box of a light switch positioned near the entryway. The virtual object 1516 can be positioned in relation to features in the environment”; since the virtual object is in the image frame, the guide information is based on the image position information).
While Kahle teaches that the guide information is based on image position information (see last citation) and that the common feature points are used to determine the camera’s orientation in the environment (Kahle, col 22, ln 58-61: “Images of features acquired by a camera of the wearable device provide information to the wearable device about orientation (e.g., with or without further data acquired by an IMU”), Kahle fails to explicitly teach calculating, with an imaging position calculation unit, position information of an imaging position and an imaging posture of the new image by using position information of the common feature points, which are associated, and the new image (emphasis added).
However, Finn teaches a similar guide system (Finn, para 17: “As an illustration, to visualize internal elements (e.g., studs, pipes, or conduits) hidden behind a finished wall, a 3D digital model of internal elements can be generated. The 3D digital model can be generated using a blueprint of a building which is used to construct the building”). Finn teaches a captured image (Finn, para 33: “a camera 141, which can be used to detect and capture an image of the marker 103”; see FIG. 3E, para 41: “FIG. 3E illustrates a schematic diagram representing an augmented reality image shown on a display screen 345 of a mobile device”), position information of common feature points (Finn, marker, para 32: “marker 103 can be placed at precise, surveyed coordinates on the external element in the real environment using the surveying equipment 112. Thus, the marker placed on the external element 102 can be used as an anchor point for augmented reality visualization of the 3D digital model of the internal elements 101”), and calculating, with an imaging position calculation unit, position information of an imaging position and an imaging posture of the new image by using position information of the common feature points, which are associated, and the new image (Finn, position and orientation of the displayed scene based on the marker orientation, para 81: “As the user walks around the project site or tilts the mobile device, the plane (i.e., physical surface) at which the physical marker 103 is affixed can change its orientation with the camera image plane. In an embodiment, when such a change is detected by the mobile device, a matrix transformation may be performed by the processor of the mobile device, and the scene seen through the mobile device can be adjusted to match the change in the mobile device position and orientation”).
Kahle discloses a base method for determining the position and orientation of a images in the environment in order to display virtual objects in relation to the displayed image frame, but does not specify specific methods for using common feature points to do so. Finn teaches a known technique of utilizing markers (targets) to determine the position and posture of a captured image. A person having ordinary skill in the art, before the effective filing date of the claimed invention, could have applied the known technique, as taught by Finn, in the same way to the guide method taught by Kahle and achieved predictable results of using an imaged marker to determine the position and posture of a captured image. Doing so allows for the correct visualizing of virtual objects placed in the field of view of the image (Finn, para 81: “This way, the 3D digital model of the internal elements can still be visualized in the geometrically correct orientation with respect to the real scenes as a user moves around. By changing the viewing angle of the 3D digital model of the internal elements, the separation of the internal elements may be better visualized. For example, if the internal elements were pipes underground, then viewing the underground pipes at an angle using a mobile device may allow visualization of separation and stacking of the pipes compared to viewing them straight from the top”).
Regarding claim 8, Kahle teaches a non-transitory tangible recording medium of a guide program to be installed into a guide terminal held by a worker in a guide system to assist the worker with work in a work site, the guide program causing a computer to execute steps (Kahle, col 25, ln 56-67 to col 26, ln 1: “Storage subsystem 2004 can be implemented using a local storage and/or removable storage medium, e.g., using disk, flash memory (e.g., secure digital card, universal serial bus flash drive), or any other non-transitory storage medium…storage subsystem 2004 can store one or more applications and/or operating system programs to be executed by processing subsystem 2002, including programs to implement some or all operations described above that would be performed using a computer.”; col 8, ln 41-46: “By showing the user a virtual object in relation to the environment, construction tasks can be simplified. In the example above, a construction worker can see, through the optical display 120, a virtual object (e.g., a dot) where an anchor point is to be located on a physical wall”). All further claim limitations are met and rendered obvious by Kahle in view of Finn because the steps performed in claim 8 are the same as those performed in claim 7.
Claim 2 is rejected under 35 U.S.C. 103 as being unpatentable over Kahle in view of Finn, in further view of Lagmanson (U.S. Patent No. 2020/0090405 A1).
Regarding claim 2 (dependent on claim 1), Kahle in view of Finn teaches wherein the guide terminal further comprises: a display unit for displaying a new image captured with the imaging unit (Finn, para 33: “Any real scenes seen through the camera and/or any images retrieved from a data storage 147 (or retrieved from the server computer 120 or a third party AR server) can be processed by a data processor 148 and displayed on a display screen 145 of the mobile device 140”). Kahle teaches displaying the guide information (see claim 1 rejection), but doesn’t explicitly teach displaying new images (Kahle, col 8, ln 41-46: “By showing the user a virtual object in relation to the environment, construction tasks can be simplified. In the example above, a construction worker can see, through the optical display 120, a virtual object (e.g., a dot) where an anchor point is to be located on a physical wall”). It would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to have combined the display unit of Finn in the system of Kahle in view of Finn in order to present captured images to the user for real-time use/analysis of objects in the environment (Finn, para 5: “on a display screen of the mobile device, an augmented reality image comprising a real view of the external element seen through the camera of the mobile device in real-time is overlaid with the 3D digital model of the internal elements in relation to the marker according to the relation data. This way, the 3D digital model of the internal elements is seen in a geometrically correct orientation with respect to the external element at a calculated distance in depth behind the external element in the real environment”).
Additionally, while Kahle in view of Finn teaches wherein the orientation of the captured image is determined and comparing positions of the imaging range with a position of the scheduled installation position of the work target object or the work target point, and if the scheduled installation position of the work target object or the work target point is included in the imaging range, overlappingly displaying the work target object or the work target point on a position corresponding to the scheduled installation position of the work target object or the work target point on the new image displayed on the display unit (Finn, the position and posture of the image frame includes the range of the image, para 32: “The marker 103 can be placed at precise, surveyed coordinates on the external element in the real environment using the surveying equipment 112.”; see para 35 citation – the position/orientation of the imaging range is required to properly display virtual elements; virtual elements are displayed geometrically correct, therefore elements outside of the image frame range are not displayed, para 35: “an augmented reality image comprising a real view of the external element seen through the camera in real-time, overlaid with the 3D digital model of the internal elements in relation to the marker according to the relation data. This results in the 3D digital model of the internal elements being displayed in a geometrically correct orientation with respect to the external element at a calculated distance in depth as they would exist in the real environment”), Kahle in view of Finn fails to explicitly teach: an imaging range calculation unit for calculating an imaging range of the new image in an absolute coordinate system or a coordinate system specific to the work site by using the position information of the imaging position and the imaging posture of the new image; and a display control unit for comparing positions of the imaging range with a position of the scheduled installation position of the work target object or the work target point, and if the scheduled installation position of the work target object or the work target point is included in the imaging range, overlappingly displaying the work target object or the work target point on a position corresponding to the scheduled installation position of the work target object or the work target point on the new image displayed on the display unit (emphasis added).
However, Lagmanson teaches a similar system (Lagmanson, para 5: “Annotation content may be rendered on a display screen of the augmented reality device that shows a user where target coordinates within the physical environment are located”), including an imaging range calculation unit for calculating an imaging range of the new image in an absolute coordinate system or a coordinate system specific to the work site by using the position information of the imaging position and the imaging posture of the new image (Lagmanson, para 51: “block 412 where image frame coordinates of the physical environment within the image frame are determined. In an embodiment of block 412, the augmented reality object positioning controller 204 and/or 304 may calculate coordinates of the physical environment 103 that are within the image frame. For example, when an image frame of the physical environment 103 is captured by the imaging sensor 224, the augmented reality device coordinates and the orientation information may be associated with the image frame at the time the image frame was captured”); and a display control unit for comparing positions of the imaging range with a position of the scheduled installation position of the work target object or the work target point, and if the scheduled installation position of the work target object or the work target point is included in the imaging range, overlappingly displaying the work target object or the work target point on a position corresponding to the scheduled installation position of the work target object or the work target point on the new image displayed on the display unit (Lagmanson, para 52: “a determination is made as to whether any of the image frame coordinates correspond with a target coordinate of a location of interest. In an embodiment of decision block 414, the augmented reality object positioning controller 204 and/or 304 may determine whether any of the image frame coordinates correspond with any of the target coordinates of the LOIs 220 and/or 312”; target coordinates correspond to the scheduled installation position, para 53: “the annotation content 222 and/or 314 may include a graphic element such as a bouncing pin, an arrow, a sphere, a bull's-eye, and/or some other visual cue at the image frame coordinate on the ground that corresponds with the target coordinate of the physical environment 103. In other embodiments, the annotation content 222 and/or 314 rendered may include instructions such as a type of object to be placed at the target coordinates, the target coordinates, directions to the target coordinate, and/or any other annotation content that may be apparent to one of skill in the art in possession of the present disclosure”).
Kahle in view of Finn discloses a system that compares positions of the imaging range to overlappingly displaying objects, but does not explicitly disclose calculating an imaging range of the new image in an absolute coordinate system or a coordinate system specific to the work site by using the position information of the imaging position and the imaging posture of the new image. Lagmanson discloses the claimed limitations, as described above.
Thus, Kahle in view of Finn and Lagmanson each disclose steps for determining when a virtual object is located within the boundaries of a captured image frame based on its position. A person of ordinary skill in the art, before the effective filing date of the claimed invention, would have recognized that the use of an image’s absolute position in the environment, as taught by Lagmanson, could have been substituted for the processing steps taught by Kahle in view of Finn (para 32-35 of Finn, cited above) because both serve the purpose of accurately displaying virtual objects. Furthermore, a person of ordinary skill in the art would have been able to carry out the substitution. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to substitute the processing steps of Kahle in view of Finn for the processing steps taught by Lagmanson above according to known methods to yield the predictable result of displaying virtual objects realistically based on the orientation and position of a captured image.
Claim 3 is rejected under 35 U.S.C. 103 as being unpatentable over Kahle in view of Finn, in further view of Lagmanson and France et al. (U.S. Patent No. 2011/0066375 A1), hereinafter France.
Regarding claim 3 (dependent on claim 2), Kahle in view of Finn and Lagmanson teaches wherein the display control unit uses the position information of the scheduled installation position of the work target object or the work target point and position information of a given point in the imaging range of the new image in the absolute coordinate system or the coordinate system specific to the work site (Taught in claims 1 and 2 rejections above). The system of Kahle in view of Finn and Lagmanson must compare the position of a given point in the imaging range and the scheduled installation position of the work target object or the work target point in order to determine which virtual objects should be rendered within the image frame and which virtual objects lie outside the image frame. However, Kahle in view of Finn and Lagmanson fails to explicitly teach to calculate a difference between the given point in the imaging range and the scheduled installation position of the work target object or the work target point, and based on the difference, outputs, to the worker, the guide information to guide from the imaging position of the new image to the scheduled installation position of the work target object or the work target point.
However, France teaches a similar system (France, abstract: “providing navigational information associated with locations of objects includes an imaging device configured to acquire image data, a visual display coupled to the imaging device and configured to display the image data”). France discloses to calculate a difference between a given point in the imaging range and the scheduled installation position of the work target object or the work target point, and based on the difference, outputs, to the worker, the guide information to guide from the imaging position of the new image to the scheduled installation position of the work target object or the work target point (France, difference between a point in the image and target point 842 or 940, which determines if the target point is outside the field of view, see FIGs. 8-9; para 66: “The point 940, however, is outside the field of view of the imaging device. In this example, apparatus 900 provides a navigational graphic 944 on the visual display 926 indicating a direction to the location of the point 940. The navigational graphic 944 may indicate to the operator a direction in which to look or walk in order to locate the real object or point. In an embodiment, the image coordinates associated with the location of the object are continuously determined and the navigational graphic is updated accordingly such that if the imaging device were panned to the right, the navigational graphic 944 would transition seamlessly as the point 940 enters the field of view of the imaging device”). It would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to have combined the navigation guidance based on position difference taught by France with the system and position information values determined by Kahle in view of Finn and Lagmanson in order to provide the user with navigational directions guiding them to an area of interest (France, para 62-66).
PNG
media_image1.png
599
461
media_image1.png
Greyscale
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over Kahle in view of Finn, in further view of Lagmanson and Kjallstrom et al. (U.S. Patent No. 2020/0160607 A1), hereinafter Kjallstrom.
Regarding claim 4 (dependent on claim 2), Kahle in view of Finn and Lagmanson fails to explicitly teach wherein the display control unit suggests, on the display unit, the work target object or the work target point, which should be selected by the worker in the selection unit, based on the position information of the imaging position of the new image.
However, Kjallstrom teaches a similar system (Kjallstrom, abstract: “To facilitate virtual training, employee on-boarding, knowledge transfer and remote assistance, embodiments provide a digital note viewable in a virtual environment which may be attached to a physical machine and viewable when a user is engaged with the system”) wherein the display control unit suggests, on the display unit, the work target object or the work target point, which should be selected by the worker in the selection unit (Kjallstrom, The system displays selectable objects, digital notes, associated with work target objects. The display of a selectable object suggests that those virtual objects should be selected; para 33: “The digital note 150 in FIG. 4 may be generated by selecting an option from a virtual menu 140 which draws a pointer from the hand-held controller 120 at the serpentine belt 130 and one of its pulleys. The user may dictate a brief description of the issue (for example, text input via speech recognition), along with a recorded audio clip 160 of the noise. The user triggers a command in the system to virtually attach the digital note 150 to the physical engine 170”; when selected, guidance information is displayed, para 25: “when the user performs the task on the machine in the physical world, the digital note may be retrieved from the system and virtually attached to the machine. The user is able to see the digital note virtually attached to the machine in the physical world using for example, a type of augmented reality (AR) device”; para 26: “An icon may be displayed 90 on the anchor point which can be selected by the user. Selection of the icon may generate the stored digital information as a digital note to the user”), based on the position information of the imaging position of the new image (Kjallstrom, As described in para 33 above, the option to select the digital note depends on the work target object being in view. Therefore, the suggestion is based on the position of the image since the digital note for an object out of frame will not be suggested.).
Prompting the user with a limited number of selections relevant to the content in the field of view of a current image reduces the mental load placed on the operator while performing tasks in the environment. It would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to have combined the display control unit suggestion, as taught by Kjallstrom above, with the system of Kahle in view of Finn and Lagmanson in order to provide selectable options relevant to the object in view (See citations from Kjallstrom in the last paragraph).
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over Kahle in view of Finn, in further view of Jiang et al. (U.S. Patent No. 2016/0269631 A1), hereinafter Jiang.
Regarding claim 6 (dependent on claim 1), Kahle in view of Finn fails to explicitly teach wherein the guide terminal further comprises a voice unit for issuing a voice, and the voice unit outputs the guide information output by the guide calculation unit with a voice.
However, Jiang similarly teaches a guide system (Jiang, para 42, 48, 69) wherein the guide terminal further comprises a voice unit for issuing a voice, and the voice unit outputs the guide information output by the guide calculation unit with a voice (Jiang, para 54: “The explanation of solving the problem is performed by displaying the instruction detail 1e and voice communication by the audio stream”). It would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to have combined the voice unit of Jiang with the system of Kahle in view of Finn in order to further aid the user while performing the guided action (Jiang, para 55: “When the operator 2 easily comprehends and confirms the operation procedure, the operator 2 performs an operation at the work site”; see also para 42).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
U.S. Patent No. 2017/0206712 A1
U.S. Patent No. 2017/0337743 A1
U.S. Patent No. 2019/0094021 A1
U.S. Patent No. 2021/0142576 A1
U.S. Patent No. 2022/0172406 A1
U.S. Patent No. 2020/0286289 A1
U.S. Patent No. 2023/0351541 A1
Any inquiry concerning this communication or earlier communications from the examiner should be directed to EMMA E DRYDEN whose telephone number is (571)272-1179. The examiner can normally be reached M-F 9-5 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, ANDREW BEE can be reached at (571) 270-5183. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/EMMA E DRYDEN/Examiner, Art Unit 2677
/ATIBA O FITZPATRICK/Primary Examiner, Art Unit 2677
Prosecution Insights