Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The amendment filed on 3/17/26 has been entered and made of record. Claim 1 is amended. Claims 1-9 are pending.
Response to Arguments
Applicant’s arguments with respect to claim 1 have been fully considered but they are moot because the arguments do not apply to the references being used in the current rejection.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(d):
(d) REFERENCE IN DEPENDENT FORMS.—Subject to subsection (e), a claim in dependent form shall contain a reference to a claim previously set forth and then specify a further limitation of the subject matter claimed. A claim in dependent form shall be construed to incorporate by reference all the limitations of the claim to which it refers.
The following is a quotation of pre-AIA 35 U.S.C. 112, fourth paragraph:
Subject to the following paragraph [i.e., the fifth paragraph of pre-AIA 35 U.S.C. 112], a claim in dependent form shall contain a reference to a claim previously set forth and then specify a further limitation of the subject matter claimed. A claim in dependent form shall be construed to incorporate by reference all the limitations of the claim to which it refers.
Claim 6 is rejected under 35 U.S.C. 112(d) or pre-AIA 35 U.S.C. 112, 4th paragraph, as being of improper dependent form for failing to further limit the subject matter of the claim upon which it depends, or for failing to include all the limitations of the claim upon which it depends. Claim 6 recites the similar limitation as its dependent claim 1.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-9 are rejected under 35 U.S.C. 103 as being unpatentable over Hara (JP2018056845A) in view of Kuroki et al. (US 2007/0002037 A1).
As to Claim 1, Hara teaches A work assistance system comprising:
a first system used by a first user (Hara discloses a worker-side device 12A in Fig 1); wherein the first system includes:
an image capturing unit that captures an image in the space (Hara discloses “The worker-side device 12A includes an imaging unit 14 that captures images of the worker's surroundings” in [0013]);
a virtual object generation unit that generates a work assistance virtual object that assists the work in the space; a first display unit that displays the work assistance virtual object superimposed on the space (Hara discloses “The worker side device 12A (worker side HMU 40A) receives data of a forward image on which instruction information is superimposed from the instructor side device 22A, and displays the forward image on which the instruction information represented by the received data is superimposed on the HMD 52… but the video see-through type is preferred as it can accurately superimpose instruction information at the appropriate position on the display screen” in [0024]); and
a first communication unit that outputs image capturing position data indicating an image capturing position of the image capturing unit in the space and the image (Hara discloses “a communication unit 18 that communicates wirelessly with the instructor-side device 22A, and an information output unit 20 that outputs information to the worker” in [0013]; “In one embodiment, the acquisition unit acquires information representing the posture of the worker's head” in [0008], see also [0015]. Here, the position of worker also refers to image capturing position data),
a second system used by a second user who gives an instruction about work in a space where the first user is present (Hara discloses an instruction-side device 22A in [0014]), the second system includes:
a second display unit that displays the image (Hara discloses “the display unit 26 displays the surrounding image generated by the image generation unit 30” in [0017]);
a data generation unit that generates instruction reference position data indicating an instruction reference position corresponding to the image capturing position data and instruction position data indicating an instruction position based on an operation of designating the instruction position on the image (Hara discloses “The instruction generation unit 36 detects the position of the work target based on the operation of the instructor notified by the operation detection unit 32, and compares the position of the work target with the visual position corresponding to the posture of the worker's head represented by the information acquired by the acquisition/communication unit 24” in [0018]); and
a second communication unit that outputs the instruction reference position data and the instruction position data, and the virtual object generation unit generates the work assistance virtual object based on the instruction reference position data and the instruction position data (Hara discloses “Furthermore, when the work target position and the worker's visual position differ, the instruction generation unit 36 determines a movement path that maintains the change in the worker's head posture within the range of motion of the head when the worker turns his/her head from the current visual position to the work target position. The instruction generating unit 36 then generates instruction information that instructs the user to perform the action along the determined movement path, and superimposes the generated instruction information on the forward image generated by the image generating unit 30” in [0018].)
Hara is silent on a line segment extending from the instruction reference position. The combination of Kuroki further teaches following limitation:
the virtual object generation unit generates the work assistance virtual object extending from the instruction reference position to the instruction position based on the instruction reference position data and the instruction position data (Kuroki discloses “The three-dimensional CG drawing devices 102 and 1502 draw respective line segment images each starting from the position of the pointing device 1506 acquired in the step S41 and ending at a part of the coordinate values acquired from the input device 105 and send the resultant CG images to the respective image synthesizing devices 103 and 1503” in [0105]; “As a result, the HMD wearer can recognize the position indicated by the non-HMD wearer, and both the HMD wearer and the non-HMD wearer can recognize the instruction information (i.e. the lead line 602, the annotation character object 603, a locus object 1901, and a line segment object 2001 extending to the pointer object)” in [0107]; “the line segment object 2001 starts from the position of the pointing device 1506 (see FIG. 21), the present invention is not limited to this, but the line segment object 2001 may start from the focal position of the image pickup section 301 of the HMD 101 (see FIGS. 24 and 25) or may start from the focal position of the objective viewpoint camera 1501 (see FIGS. 26 and 27)” in [0112], see also Fig 24-25.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the invention of Hara with the teaching of Kuroki so that the HMD wearer can recognize the position indicated by the non-HMD wearer, and both the HMD wearer and the non-HMD wearer can recognize the instruction information (Kuroki, [0107]).
As to Claim 2, Hara in view of Kuroki teaches The work assistance system according to claim 1, wherein when each of the operations of designating the instruction position on each of a plurality of the images with the image capturing position data different from each other is an operation of designating the same instruction position, the data generation unit generates a plurality of combined data of the instruction reference position data and the instruction position data in response to each of the operations, and the virtual object generation unit calculates a first intersection point of straight lines passing through positions indicated by the instruction reference position data and the instruction position data of each of the plurality of combined data, and generates the work assistance virtual object based on the first intersection point (Hara discloses “In other words, the instruction information is information that instructs the worker to turn his /her head from the current visual position to the target visual position along a movement path that maintains the change in posture of the worker's head during the movement within the range of motion of the head.” in [0008]; “FIG. 10 is an image diagram of an example of a work instruction superimposed on a forward image” in [0010]. Kuroki further discloses “Although in the fourth embodiment described above, in step S43, the pointer object, the lead line 602 starting from the position of the pointer object, the annotation character object 603 at the end of the lead line 602, the CG images generated by texture mapping, and the line segment object 2001 (hereinafter the pointer object, the lead line 602, and the annotation character object 603 will be collectively referred to as the instruction objects) are all displayed as instruction objects on the displays 104 and 1504, the present invention is not limited to this, but only necessary ones of the instruction objects may be selectively displayed on the displays 104 and 1504” in [0110], see also [0112] and Fig 20-27.)
As to Claim 3, Hara in view of Kuroki teaches The work assistance system according to claim 2, wherein the virtual object generation unit generates, as the work assistance virtual object, a virtual object along a line segment extending from the instruction reference position indicated by the instruction reference position data of each of the plurality of combined data to the first intersection point (Kuroki, Fig 20-27.)
As to Claim 4, Hara in view of Kuroki teaches The work assistance system according to claim 3, wherein the virtual object generation unit generates a virtual mark placed at the first intersection point as the work assistance virtual object (Hara discloses “This allows the instructor to grasp the overall situation at the work site by visually checking the surrounding image displayed on the display unit 26, identify the location where the worker will be working, and perform a work target location designation operation to specify the identified work target location on the surrounding image via the input unit 28” in [0036]; “where the operation detection unit 32 determines whether or not it has detected that the instructor has performed a work target position designation operation” in [0038]; “in which the operation detection unit 32 generates a designation operation notification indicating that a work target position designation operation has been performed… The work target position information includes data on the peripheral image (hereinafter referred to as the "specified operation image") that was displayed on the display unit 26 at the time the instructor specified the work target position on the peripheral image, and the two-dimensional coordinate values of the work target position on the specified operation image” in [0039]; “In step 166, the operation detection unit 32 presents the contents of the generated notification to the instructor by displaying them on the display unit 26” in [0042]. Kuroki, Fig 20-27.)
As to Claim 5, Hara in view of Kuroki teaches The work assistance system according to claim 1, wherein the first system further includes a scan unit that generates three-dimensional scan data about a real object in the space by scanning in the space, and the virtual object generation unit calculates a second intersection point of an outer surface of the real object based on the three-dimensional scan data and a straight line passing through positions indicated by the instruction reference position data and the instruction position data, and generates the work assistance virtual object based on the second intersection point (Hara discloses “The worker side HMU 40A includes a CPU 42, a memory 44, a storage unit 46, a plurality of head-mounted cameras (HMC) 48 which are an example of the imaging unit 14” in [0019]; “For this reason, the multiple HMCs 48, which are an example of the imaging unit 14, are cameras with the same angle of view, and are attached to the worker's head with their respective imaging directions adjusted so that they can capture images of the worker's surroundings in all directions without any gaps” in [0020]. It is well-known that the HMD with stereo cameras can generate 3D model data. Hara also discloses “and when a specified visual target position differs from a visual position corresponding to the current posture of the head of the worker, generating instruction information for instructing the worker to turn his/her head from the current visual position to the visual target position along a movement path that maintains a change in the posture of the head of the worker within a range of motion of the head.” in [0114]. Kuroki discloses “In the case where the right and left display sections of the HMD 101 display images with different parallaxes, the HMD wearer can stereoscopically recognize the position of a pointer object 601 (see FIG. 6; described later in detail) from a three-dimensional perspective.” in [0064], see also Fig 20-27.)
As to Claim 6, Hara in view of Kuroki teaches The work assistance system according to claim 5, wherein the virtual object generation unit generates, as the work assistance virtual object, a virtual object along a line segment extending from the instruction reference position indicated by the instruction reference position data to the second intersection point (Kuroki, Fig 20-27.)
As to Claim 7, Hara in view of Kuroki teaches The work assistance system according to claim 5, wherein the virtual object generation unit generates a virtual mark placed at the second intersection point as the work assistance virtual object (Hara discloses “in which the operation detection unit 32 generates a designation operation notification indicating that a work target position designation operation has been performed… The work target position information includes data on the peripheral image (hereinafter referred to as the "specified operation image") that was displayed on the display unit 26 at the time the instructor specified the work target position on the peripheral image, and the two-dimensional coordinate values of the work target position on the specified operation image” in [0039]; “In step 166, the operation detection unit 32 presents the contents of the generated notification to the instructor by displaying them on the display unit 26” in [0042]. Kuroki, Fig 20-27.)
As to Claim 8, Hara in view of Kuroki teaches The work assistance system according to claim 1, wherein when the operation of designating the instruction position is continuously performed on the image, the data generation unit generates a plurality of the continuous instruction position data in response to the operation, and the virtual object generation unit generates, as the work assistance virtual object, a plurality of virtual objects along a plurality of straight lines extending from the instruction reference position indicated by the instruction reference position data toward the instruction position indicated by each of the plurality of instruction position data (Hara discloses “In the next step 106, the worker-side device 12A determines whether a "movement completion display" indicating that the movement of the viewpoint has been completed is superimposed on the forward image represented by the data received from the instructor-side device 22A. If the determination in step 106 is negative, the process returns to step 100, and steps 100 to 106 are repeated until the determination in step 106 is positive. Therefore, during this time, information indicating the posture of the worker's head and captured image data are repeatedly transmitted, and data of the forward image with instruction information superimposed is received and displayed on HMD 52, and the worker performs actions such as moving the viewpoint to the work target position in accordance with the instructions” in [0032]. Kuroki, Fig 20-27.)
As to Claim 9, Hara in view of Kuroki teaches The work assistance system according to claim 1, further comprising a server device connected to the first communication unit and the second communication unit via a communication network to allow data communication, wherein the server device includes a server storage unit that stores the image capturing position data, the image, the instruction reference position data, and the instruction position data (Hara discloses a communication between a worker-side device 12A and an instructor-side device 22A as shown in Fig 1. It is obvious that a server device to connect the worker-side device 12A and an instructor-side device 22A.)
Conclusion
THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WEIMING HE whose telephone number is (571)270-1221. The examiner can normally be reached Monday-Friday, 8:30am-5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Tammy Goddard can be reached on 571-272-7773. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Weiming He/
Primary Examiner, Art Unit 2611
Prosecution Insights