DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Status
Claims 1-4, 6-11, and 13-18 have been amended. Claims 1-18 remain pending and are ready for examination.
Claim Rejections - 35 USC § 112
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that use the word “means” or “step” but are nonetheless not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph because the claim limitation(s) recite(s) sufficient structure, materials, or acts to entirely perform the recited function. Such claim limitation(s) is/are: “the executable program logic is configured to” in claim 1, 3-9, 11-12, 14-16, and 18.
Because this/these claim limitation(s) is/are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are not being interpreted to cover only the corresponding structure, material, or acts described in the specification as performing the claimed function, and equivalents thereof.
If applicant intends to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to remove the structure, materials, or acts that performs the claimed function; or (2) present a sufficient showing that the claim limitation(s) does/do not recite sufficient structure, materials, or acts to perform the claimed function.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1, 3, 10-13, and 17 is/are rejected under 35 U.S.C. 102(a)(1) as anticipated by Ha et al. (US20210201584A1 -hereinafter Ha).
Regarding Claim 1, Ha teaches:
An automation system for controlling an automated production process of an industrial plant (see Abstract; Ha: “Provided is a method for monitoring a field based on augmented reality using a digital twin performed in a processor of a computing device of a manager located remotely from a physical space of a field”), the automation system comprising:
an augmented reality system (see [0011]; Ha: “an augmented reality environment”), comprising:
augmented reality glasses (see [0117]; Ha: “The computing device 100 according to an embodiment may include a wearable type computing device 100 such as a smart glasses display or a head mounted display (HMD).”); and
executable program logic coupled to the augmented reality glasses (see [0132]; Ha: “The processor assembly 120 may include at least one processor capable of executing instructions of the augmented reality application 111 stored in the memory 110 in order to perform various works for creating the augmented reality environment.”), the executable program logic configured to:
receive an up-to-date spatial position of a user wearing the augmented reality glasses from a positioning system, (see [0014]; Ha: “acquiring the location information of the field user for the physical space based on the positioning sensor data received from the field sensor system arranged in the field; acquiring the positioning sensor data acquired by receiving a UWB radio signal generated from a tag of the field user by a plurality anchors.”)
dynamically determine coordinates of a bounding space proximate to the user wearing the AR-glasses, (see [0014]; Ha: “tracking real-time actual coordinates of the physical space for the field user based on the acquired positioning sensor data;”)
display one or more virtual objects to the user wearing the AR glasses via the AR glasses if and only if coordinates of the virtual objects are within the bounding space (see [0014]; Ha: “calculating virtual coordinates of the three-dimensional virtual space matched to the actual coordinates; and displaying the first virtual content representing identification information of the field user on the calculated virtual coordinates.”), wherein the displayed virtual objects comprise data associated with monitoring and/or controlling the automated production process, (see [0041]; Ha: “The virtual contents as virtual contents generated by a computing device include labels, text information, images, drawing objects, and 3D entities, and are augmented/mixed contents generated in response to the physical space or an image acquired by capturing the physical space.” See [0043]: “Furthermore, the remote manager means a person who monitors a field situation through the field twin model at a remote location from a field in which the work is performed and provides a work guidance for instructing or assisting the work based on the augmented reality communication to the field user as the virtual contents.”)
continuously update the coordinates of the bounding space and the virtual objects displayed therein as a function of the up-to-date spatial position of the user wearing the AR glasses. (see [0232]; Ha: “As such, the monitoring application 311 or 611 may generate and provide a field twin model that updates the real object information and the field user information to the virtual space matched to the actual field physical space 10 in real time and provide a monitoring function to monitor an overall field and check respective detailed information by the remote manager.”)
Regarding Claim 3, Ha teaches all the limitations of claim 1 above, Ha further teaches wherein the executable program logic is configured to dynamically determine the coordinates of the bounding space in dependence on a number of equipment and/or a corresponding number of virtual objects comprising data of said equipment in spatial proximity to the user wearing the AR-glasses. (see [0041]; Ha: “Here, an augmented reality environment may mean an environment in which virtual contents associated with a physical space around a user or a communication counterpart are inserted and generated, and as a result, information provided to the user is augmented and include a mixed reality in addition to the augmented reality (AR) environment. The virtual contents as virtual contents generated by a computing device include labels, text information, images, drawing objects, and 3D entities, and are augmented/mixed contents generated in response to the physical space or an image acquired by capturing the physical space.”)
Regarding Claim 10, Ha teaches all the limitations of claim 1 above, Ha further teaches wherein at least some of the virtual objects are graphical representations of a respective one of the equipment represented by the virtual object (see [0120]; Ha: “For example, referring to FIG. 4, an augmented reality application 111 of the computing device 101 may perform image recognition for a real object RO and a marker MK in the real object in the surrounding physical space 10 and control a virtual content VC1 to be displayed on the view field of the user corresponding to the recognized marker MK.”), wherein a graphical representation comprises an image-based illustration and/or a video-based illustration of the one equipment. (see [0122]-[0123]; Ha: “In addition, the augmented reality application 111 may recognize the physical space through the trained real object or marker MK and display a virtual content matched to a specific location of the recognized space to correspond to the view field of the user. The virtual content may include an image or a video which may be displayed at a portion of the view field of the user in the computing device. For example, the virtual content may include virtual object images that overlay various portions of the physical space. The virtual object image may be rendered as a 2D image or 3D image.”)
Regarding Claim 11, Ha teaches all the limitations of claim 1 above, Ha further teaches wherein the bounding space is one of two or more data-type specific bounding spaces (see [0120]; Ha: “For example, referring to FIG. 4, an augmented reality application 111 of the computing device 101 may perform image recognition for a real object RO and a marker MK in the real object in the surrounding physical space 10 and control a virtual content VC1 to be displayed on the view field of the user corresponding to the recognized marker MK.”. See [0105]-[0106]: “the field twin model 50 may display real-time location information of the field user as a virtual content (VR) based on the sensor data of the field sensor system 400 and the virtual content (VR) may further include a work history, work information, etc., input by the field user or confirmed through the sensor data.”) [That is, the virtual content reads on ‘the bounding space’. virtual content displayed on VR glasses and the virtual content displayed on the remote computing device are two data-type specific bounding spaces], wherein the virtual objects comprise objects of data types selected from a group consisting of videos, images, texts, audio files and alarm messages (see [0041]; Ha: “The virtual contents as virtual contents generated by a computing device include labels, text information, images, drawing objects, and 3D entities, and are augmented/mixed contents generated in response to the physical space or an image acquired by capturing the physical space.”), and wherein the executable program logic is configured to dynamically determine the coordinates of a data-type specific bounding space in dependence on the data types of the virtual objects to be displayed to the user wearing the AR-glasses. (see [0041]; Ha: “Here, an augmented reality environment may mean an environment in which virtual contents associated with a physical space around a user or a communication counterpart are inserted and generated, and as a result, information provided to the user is augmented and include a mixed reality in addition to the augmented reality (AR) environment”)
Regarding Claim 12, Ha teaches all the limitations of claim 1 above, Ha further teaches wherein the executable program logic is configured to determine an alarm bounding space for the alarm messages (see [0278]; Ha: “The monitoring application 311 or 611 may highlight the virtual content representing the location of the field user entering the dangerous zone in the field twin model 50 and notify the remote manager of the entrance of the field user into the dangerous area.”) and at least a further bounding space for one of the other data types (see [0316]; Ha: “For example, referring to FIG. 17, when viewing an actual space including the dangerous area through the wearable type computing device 100, an augmented reality environment to which a red translucent 30-th virtual image (VR30) indicating that there is a risk of electric shock in the area corresponding to the actual coordinates for the dangerous area in the actual space is overlaid may be provided to the field user.”), the alarm bounding space being larger than the further bounding space. (see [0314] and Fig. 14; Ha: “the augmented reality application 111 may convert the virtual 3D spatial coordinates into the actual spatial coordinates of the physical space 10 in the field to acquire the 3D actual coordinates for the dangerous area of the physical space 10.” See [0316] and Fig. 17: “when viewing an actual space including the dangerous area through the wearable type computing device 100, an augmented reality environment to which a red translucent 30-th virtual image (VR30) indicating that there is a risk of electric shock in the area corresponding to the actual coordinates for the dangerous area in the actual space is overlaid may be provided to the field user.”) [That is, the 3D actual coordinates for the dangerous area of the field twin model 50 reads on ‘the alarm bounding space’ and the virtual image corresponding to the actual coordinates for the dangerous area through the wearable type computing device reads on ‘the further bounding space’. Therefore, the the 3D actual coordinates for the dangerous area of the field twin model 50 is larger than the virtual image corresponding to the actual coordinates for the dangerous area through the wearable type computing device.]
Regarding Claim 13, Ha teaches all the limitations of claim 1 above, Ha further teaches wherein
the bounding space surrounds the user wearing the AR glasses (see [0118]; Ha: “A smart glass type computing device 101 may include a display system including glasses displaying the virtual content (e.g., virtual object image) on a view field of a user while transmitting light so as for the user to view the surrounding physical space while being worn.”); or
wherein the bounding space is in the field of view of the user wearing the AR glasses and does not surround the user wearing the AR glasses. (see [0103]; Ha: “Referring to FIG. 3, as an example, the remote computing device 301: 300 and 600 may generate and output a virtual field twin model 50 corresponding to a physical space 10 of an actual field.”)
Regarding Claim 17, the limitations in this claim is taught by Ha as discussed connection with claim 1.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 2, 15, and 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ha in view of Schmirler et al. (US20200336706A1 -hereinafter Schmirler).
Regarding Claim 2, Ha teaches all the limitations of claim 1 above, Ha further teaches wherein coordinates of each one of the plurality of equipment defines the coordinates of at least one of the virtual objects comprising the data enabling the user to operate and/or maintain the one equipment. (see [0300]; Ha: “In addition, in the augmented reality application 111, the received virtual content is matched to actual coordinates of the field physical space 10 matched to the three-dimensional virtual coordinates and then, the three-dimensional virtual coordinates are augmented (e.g., inserted) into the actual coordinates and displayed to intuitively check the work instruction or guidance of the remote manager through the augmented reality communication.”)
However, Ha does not explicitly teach: wherein the industrial plant includes a plurality of equipment for performing the production process, the plurality of equipment being spatially distributed in the industrial plant, wherein the virtual objects include graphical and/or acoustic representations of data enabling the user to operate and/or maintain the plurality of equipment,
Schmirler from the same or similar field of endeavor teaches wherein the industrial plant includes a plurality of equipment for performing the production process (see [0004]; Schmirler: “In one or more embodiments, a system is provided, comprising a device interface component configured to receive industrial data from industrial devices associated with an industrial facility.”), the plurality of equipment being spatially distributed in the industrial plant (see [0072]; Schmirler: “In some scenarios, the industrial devices may be distributed across multiple plant networks 116 within the plant facility.”), wherein the virtual objects include graphical and/or acoustic representations of data enabling the user to operate and/or maintain the plurality of equipment, (see [0110]; Schmirler: “The virtual view can include graphical representations of the devices, as well as relevant data associated with one or more of the devices contained within the cabinet (obtained from plant data 610). In some embodiments, the user can send a request to present system 302 (e.g., via a gesture or verbal command recognizable to the wearable appliance) for additional information about the control cabinet, including electrical schematics or line diagrams for the cabinet, ladder logic programming associated with an industrial controller mounted within the cabinet, diagnostic data for any of the devices, etc.”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to modify the teaching of Ha to include Schmirler’s features of the industrial plant includes a plurality of equipment for performing the production process, the plurality of equipment being spatially distributed in the industrial plant, wherein the virtual objects include graphical and/or acoustic representations of data enabling the user to operate and/or maintain the plurality of equipment. Doing so would guide users through the process of correcting detected maintenance issues. (Schmirler, Abstract)
Regarding Claim 15, Ha teaches all the limitations of claim 1 above; however, Ha does not explicitly teach wherein the executable program logic is configured to receive data being indicative of an orientation of the AR glasses and is configured to dynamically determine and continuously update the coordinates of the bounding space also as a function of the data being indicative of the orientation of the AR glasses.
Schmirler from the same or similar field of endeavor teaches wherein the executable program logic is configured to receive data being indicative of an orientation of the AR glasses (see [0070]; Schmirler: “Location and orientation component 410 can be configured to determine a location and an orientation of the wearable appliance 206.”) and is configured to dynamically determine and continuously update the coordinates of the bounding space also as a function of the data being indicative of the orientation of the AR glasses. (see [0092]; Schmirler: “As the user moves around, toward, or away from the virtual scaled industrial area, the wearable appliance 206 streams updated location and orientation data 606 to the presentation system 302, which updates the presentation data 604 substantially continuously to simulate the effect of walking around a scale model of the production area.”)
The same motivation to combine Ha and Schmirler a set forth for Claim 2 equally applies to Claim 15.
Regarding Claim 18, the limitations in this claim is taught by the combination of Ha and Schmirler as discussed connection with claim 15.
Claim(s) 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ha in view of Linden (US20190130297A1 -hereinafter Linden).
Regarding Claim 16, Ha teaches all the limitations of claim 1 above; however, it does not explicitly teach wherein the executable program logic is configured to receive data being indicative of an eye orientation of the user wearing the AR glasses and is configured to dynamically determine and continuously update the coordinates of the bounding space as a function of the eye orientation of the user wearing the AR glasses.
Linden from the same or similar field of endeavor teaches wherein the executable program logic is configured to receive data being indicative of an eye orientation of the user wearing the AR glasses (see [0047]; Linden: “The device may e.g. be a virtual reality headset or a pair of augmented reality glasses. The device may comprise at least one illuminator for illuminating an eye of the user, at least one camera for capturing images of the eye, and a circuitry.” See [0056]: “The data set is not only more complete but it is also considerate of context, e.g., specific to the subject and the current conditions such as ambient light and orientation of eye tracking system with respect to the subject.”) and is configured to dynamically determine and continuously update the coordinates of the bounding space as a function of the eye orientation of the user wearing the AR glasses. (see [0088]; Linden: “On the contrary, the compensation model may be formed based on eye tracking data obtained from three or more images, and may further be dynamically updated during use of the eye tracking system.”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to modify the teaching of Ha to include Linden’s features of receiving data being indicative of an eye orientation of the user wearing the AR glasses and dynamically determining and continuously updating the coordinates of the bounding space as a function of the eye orientation of the user wearing the AR glasses. Doing so would improve the accuracy of the eye tracking. (Linden, [0036])
Claim(s) 4-6 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ha in view of Zhang et al. (US20220358716A1 -hereinafter Zhang) in view of Duff et al. (US20200334395A1 -hereinafter Duff).
Regarding Claim 4, Ha teaches all the limitations of claim 1 above; however, Ha does not explicitly teach wherein the industrial plant comprises multiple different rooms and/or compartments having different heights, and wherein the executable program logic is configured to dynamically determine the coordinates, the coordinates corresponding to height, of the bounding space in dependence on the height of the room and/or compartment within which the up-to-date spatial position of the user wearing the AR-glasses is located.
Zhang from the same or similar field of endeavor teaches wherein the industrial plant comprises multiple different rooms and/or compartments having different heights, (see [0060]; Zhang: “The multiple closures are independent from each other and can have different ceiling heights. Therefore, the multiple closures can be used to represent a scene of multiple rooms with different ceiling heights.”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to modify the teaching of Ha to include Schmirler’s features of comprising multiple different rooms and/or compartments having different heights. Doing so would yield a more accurate construction and improve the quality of the generated 3D geometry. (Zhang, [0062] and [0064])
However, it does not explicitly teach: and wherein the executable program logic is configured to dynamically determine the coordinates, the coordinates corresponding to height, of the bounding space in dependence on the height of the room and/or compartment within which the up-to-date spatial position of the user wearing the AR-glasses is located.
Duff from the same or similar field of endeavor teaches and wherein the executable program logic is configured to dynamically determine the coordinates, the coordinates corresponding to height (see [0171]; Duff: “first level of geospatial location determinations may be based upon the real estate parcels 140-143 themselves and a second geospatial determination may be made according to Reference Position Transceivers (discussed more fully below) included within the boundaries of the real estate parcels 140-143.”), of the bounding space in dependence on the height of the room and/or compartment within which the up-to-date spatial position of the user wearing the AR-glasses is located. (see [0186]; Duff: “In another aspect, an Augmented Virtual Model 100 may indicate that location identifiers 121A are placed at two or more corners (or other placement) of a physical structure 102A and each of the location identifiers 121A may include a transmitter with a defined location and at a defined height. The user device 106, or other type of controller, may then triangulate with the location identifiers 121A to calculate a precise location and height within the physical structure.” See [0289]: “The dimensions of these various components may be dynamically updated based on an original model that may be compared to actual fabricated structure as realized on a building site.” See Abstract; Duff: “Methods and apparatus for presenting data to a user with augmented reality headgear or augmented reality glasses that has been oriented in a direction based upon unique automated generation of a vector are described.”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to modify the teaching of Ha and Schmirler to include Duff’s features of dynamically determining the coordinates, the coordinates corresponding to height, of the bounding space in dependence on the height of the room and/or compartment within which the up-to-date spatial position of the user wearing the AR-glasses is located. Doing so would clear presentation of model data and also improve the ability to update datasets by collection of new imagery by camera and scanning systems attached to the Headgear. (Duff, [0031])
Regarding Claim 5, the combination of Ha, Schmirler, and Druff teaches all the limitations of claim 4 above, Druff further teaches wherein the executable program logic is further configured to:
determine the height of the room or compartment by receiving measurement data from one or more sensors worn by the user; or
determine the height of the room or compartment as a function of a room-ID or compartment-ID detected by a sensor worn by the user. (see [0186]; Duff: “An AVM may indicate that one or more RFID chips are accessible in a kitchen, a living room and each bedroom of a structure. The user may activate appropriate Sensors to read the RFID chips and determine their location. In another aspect, an Augmented Virtual Model 100 may indicate that location identifiers 121A are placed at two or more corners (or other placement) of a physical structure 102A and each of the location identifiers 121A may include a transmitter with a defined location and at a defined height. The user device 106, or other type of controller, may then triangulate with the location identifiers 121A to calculate a precise location and height within the physical structure.”)
The same motivation to combine Ha, Schmirler, and Druff a set forth for Claim 4 equally applies to Claim 5.
Regarding Claim 6, Ha, Schmirler, and Druff teaches all the limitations of claim 5 above, Druff further teaches wherein the executable program logic is further configured to determine the height of the room or compartment as a function of a room-ID or compartment-ID based on an assignment table that assigns heights to room-IDs and/or compartment-IDs. (see [0186]; Duff: “An AVM may indicate that one or more RFID chips are accessible in a kitchen, a living room and each bedroom of a structure. The user may activate appropriate Sensors to read the RFID chips and determine their location. In another aspect, an Augmented Virtual Model 100 may indicate that location identifiers 121A are placed at two or more corners (or other placement) of a physical structure 102A and each of the location identifiers 121A may include a transmitter with a defined location and at a defined height. The user device 106, or other type of controller, may then triangulate with the location identifiers 121A to calculate a precise location and height within the physical structure.”) [The RFID chips reads on ‘room-ID’]
The same motivation to combine Ha, Schmirler, and Druff a set forth for Claim 4 equally applies to Claim 6.
Claim(s) 7-9 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ha in view of Duff et al. (US20200334395A1 -hereinafter Duff).
Regarding Claim 7, Ha teaches all the limitations of claim 1 above, Ha further teaches and wherein the executable program logic is configured to dynamically determine the coordinates (see [0057]; Ha: “The field monitoring server 520 may receive information on the real objects and the field users in the physical space from the field sensor system 400 or/and the computing device 100 of the field user, and update the received information to the field twin model in real time.”), the coordinates corresponding to height, size and/or shape of the bounding space in dependence on the one or more user roles assigned to the user wearing the AR-glasses. (see [0316]-[0317]; Ha: “For example, referring to FIG. 17, when viewing an actual space including the dangerous area through the wearable type computing device 100, an augmented reality environment to which a red translucent 30-th virtual image (VR30) indicating that there is a risk of electric shock in the area corresponding to the actual coordinates for the dangerous area in the actual space is overlaid may be provided to the field user. In addition, an augmented reality environment in which a 40-th virtual content (VR40) indicating the work guidance is overlaid around a real object that needs to perform the work may be provided to the field user.”)
However, Ha does not explicitly teach wherein the automation system comprises a user database and the user wearing the AR glasses is a registered user and the user database comprises an assignment of each registered user and one or more user roles,
Duff from the same or similar field of endeavor teaches wherein the automation system comprises a user database and the user wearing the AR glasses is a registered user (see [0576]; Duff: “At method step 2205, a unique identifier may be associated with the smart device and/or the Agent.”) and the user database comprises an assignment of each registered user and one or more user roles, (see [0577]; Duff: “At method step 2206, a purpose of the Agent presence within the structure is recorded. Exemplary purposes may include, one or more of: a service call placed from the structure to the system detailing an equipment malfunction; service calls from the structure detailing non-specific malfunctions and symptomatic data indicating an equipment malfunction; a service call placed by self-assessing equipment utilizing internet of things (IoT) and machine learning functionality to ascertain malfunctions and predictive analytics to anticipate malfunctions, and the like. The system may integrate data reports into the AVM and relay content of the reports to the smart device associated with the Agent in the field.”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to modify the teaching of Ha and Schmirler to include Duff’s features of a user database and the user wearing the AR glasses is a registered user and the user database comprises an assignment of each registered user and one or more user roles. Doing so would clear presentation of model data and also improve the ability to update datasets by collection of new imagery by camera and scanning systems attached to the Headgear. (Duff, [0031])
Regarding Claim 8, Ha teaches all the limitations of claim 1 above; however, Ha does not explicitly teach wherein the executable program logic is configured to dynamically determine the coordinates, the coordinates corresponding to height, size and or shape of the bounding space such that the user wearing the AR-glasses is presented virtual objects associated with equipment in multiple different rooms and/or floors of the industrial plant.
Duff from the same or similar field of endeavor teaches wherein the executable program logic is configured to dynamically determine the coordinates, the coordinates corresponding to height, size and or shape of the bounding space such that the user wearing the AR-glasses is presented virtual objects associated with equipment in multiple different rooms and/or floors of the industrial plant. (see [0105]; Duff: “In some embodiments, triangulation may calculate a position within a boundary created by the reference points to within millimeter range. In some embodiments, Differential GPS may be used to accurately determine a location of a Smart Device with a sub centimeter accuracy.” See Abstract: “Methods and apparatus for presenting data to a user with augmented reality headgear or augmented reality glasses that has been oriented in a direction based upon unique automated generation of a vector are described.” See [0186]: “Once accessed, the app may additionally search for one or more location identifiers 121A of a type and in a location recorded in the AVM. An AVM may indicate that one or more RFID chips are accessible in a kitchen, a living room and each bedroom of a structure.”)
The same motivation to combine Ha and Duff a set forth for Claim 7 equally applies to Claim 8.
Regarding Claim 9, the combination of Ha and Duff teaches all the limitations of claim 7 above, Duff further teaches wherein the user roles comprise a maintenance user role and an operation user role (see [0094]; Duff: “a technical library specific to a particular property and location within the property may be maintained for each property and made accessible to an onsite technician and/or remote expert. The library may include, but is not limited to: structure, equipment/machinery manuals; repair bulletins, and repair/maintenance. Appropriate how to videos may also be made available based upon an AVM with As Built and Experiential Data.”), and wherein the executable program logic is configured to dynamically determine the coordinates, the coordinates corresponding to height, of the bounding space such that the user wearing the AR-glasses is presented virtual objects associated with equipment in multiple different rooms and/or floors of the industrial plant if the user has a maintenance user role, but not if the user has an operator role. (see [0573]; Duff: “At method step 2202, the Agent's entry into the structure will be registered. Registration of entry into the structure may be achieved through multiple methods, which may include, by way of non-limiting example, on or more of: WiFi gateway detection, infrared detection, magnetic door locking systems, Bluetooth services, and the like. Upon entry into the structure requesting the service call, system will register the service technician's entry into the structure.” See [0579]: “At step 2207, the method steps may use one or more of: augmented reality overlays; heads-up displays or other wearable technologies; augmented reality overlays displayed on smart devices; direct instructional vectoring provided to the smart device by the system over WiFi internet connection or LTE signal; virtual reality walkthrough instructions provided to a user interface and/or the smart device on site associated with the Agent, updated map/schematic displays detailing the structure and directing the Agent to a source of a purpose for being within the structure based upon vectoring and orienteering processes.”)
The same motivation to combine Ha and Duff a set forth for Claim 7 equally applies to Claim 9.
Claim(s) 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ha in view of Chhipa et al. (US20190221191A1 -hereinafter Chhipa).
Regarding Claim 14, Ha teaches all the limitations of claim 1 above, Ha further teaches wherein the executable program logic is configured to:
receive data being indicative of the position of one or more physical objects (see [0015]; Ha: “receiving information on a real object arranged in the physical space of the field from the field sensor system arranged in the field.”), each physical object being an equipment used for the automated production process or a material used, modified or produced by the production process; (see [0214]; Ha: “The gesture input and the operation input of the guide tool may also be input into the display 671 in response to the captured image (or field twin model 50) and the computing device 600 may detect the user input by matching the real object image displayed in the captured image (or field twin model 50).” See [0215]: “That is, a monitoring application 611 of the tabletop type computing device 600 according to an embodiment may execute a series of processes for implementing a service for visualizing and providing the virtual content based on the user input acquired through the sensor system 660.”)
However, Ha does not explicitly teach:
determine the line of sight between the user wearing the glasses and one of the virtual objects;
display the one virtual object via the AR glasses if and only if the virtual object is within the bounding space and none of the physical objects is positioned in the line of sight between the user wearing the glasses and the virtual object.
Chhipa from the same or similar field of endeavor teaches
determine the line of sight between the user wearing the glasses and one of the virtual objects; (see [0084]; Chhipa: “This enhances the user's focus on only AR content in the line of sight of the user. In yet some other cases, the eye-tracking data is used to determine the focus of the user's sight towards a particular AR content which may be used to zoom in on the AR content for better viewing experience of the user.”)
display the one virtual object via the AR glasses if and only if the virtual object is within the bounding space and none of the physical objects is positioned in the line of sight between the user wearing the glasses and the virtual object. (see [0085]; Chhipa: “the eye-tracking data is also used to limit the number of AR contents to be displayed by removing the AR content which may be available in the field of view but not within the direction of the eye-gaze of the end-user.”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to modify the teaching of Ha to include Chhipa’s features of determining the line of sight between the user wearing the glasses and one of the virtual objects; and displaying the one virtual object via the AR glasses if and only if the virtual object is within the bounding space and none of the physical objects is positioned in the line of sight between the user wearing the glasses and the virtual object. Doing so would determine the best-suited AR content for viewing by the user. (Chhipa, [0006])
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Hashimoto (US12579773B2) discloses eliminating or reducing the interfering caused by the interfering object for viewing the target object.
Kim (US10575920B2) discloses the augmented reality image projection system according to at least one embodiment can be built in a compact form so that it minimizes blocked line of sight or intervened movement of the operator, providing easier manipulation in various surgical operations.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to VI N TRAN whose telephone number is (571)272-1108. The examiner can normally be reached Mon-Fri 9:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, ROBERT FENNEMA can be reached at (571) 272-2748. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/V.N.T./Examiner, Art Unit 2117
/Christopher E. Everett/Primary Examiner, Art Unit 2117
Prosecution Insights