DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 07/13/2026 has been entered.
Response to Amendment
This is in response to applicant’s amendment/response filed on 06/22/2026, which has been entered and made of record. Claim 1 has been amended. Claims 1-11 are pending in the application wherein Claims 2-7 are withdrawn.
Response to Arguments
Applicant's arguments filed on 06/22/2026 have been fully considered but they are rendered moot in view of the new grounds of rejection presented below (as necessitated by the amendment to claim 1).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1 is/are rejected under 35 U.S.C. 103 as being unpatentable over U.S. PGPubs 2016/0033770 to Fujimaki et al. in view of U.S. PGPubs 2020/0186608 to Ono et al..
PNG
media_image1.png
322
472
media_image1.png
Greyscale
Regarding claim 1, Fujimaki et al. teaches an information processing device comprising: at least one processor; and at least one memory storing at least one program that, when executed by the at least one processor, causes the information processing device to (par 0076):
acquire information indicating a current position in a three-dimensional space associated with a virtual moving object that is a fictional flying object moving aboveground over time (par 0145-0150, “In the methods c1 and c2, the AR processing unit 142a can determine an acquisition range of three-dimensional information on the basis of the current position coordinate of the user detected by the GPS module 134. In the method c3, when the three-dimensional information is generated with the camera 61, the AR processing unit 142a causes the camera 61 to acquire plural external scene images while continuously changing the angle. Then, the AR processing unit 142a can generate the three-dimensional information on the basis of difference information between the n-th (where n is an integer) captured external scene image and the (n+1)-th captured external scene image …. the AR processing unit 142a generates a mask object. The “mask object” is three-dimensional information viewed from the position and the direction corresponding to the current position and the eyeline of the user in the real space, in other words, three-dimensional information viewed from the current position and the eyeline of the virtual user. The AR processing unit 142a generates the mask object using the three-dimensional information obtained in step S302 and the positions and the directions in the three-dimensional information specified in step S304”, par 0177-0179, “The AR processing unit 142b controls behavior of the virtual object OB on the basis of the detected coordinate position and shape of the real object A. That is, it is described above that the relative coordinate position in the real space is set on the basis of the contents data, but strictly, the relative coordinate position is set on the basis of the contents data and the detected coordinate position and shape of the real object A”, par 0184, “the AR processing unit 142b sets the coordinate position (coordinate position relative to the real object A) of the virtual object OB in the real space on the basis of the contents data and the detected coordinate position and shape of the real object A (S8). The AR processing unit determines the display position, the display size, and the direction of the virtual object OB so as to visually recognize the virtual object in the coordinate system of the real space on the basis of the estimated viewing field VR, the set coordinate position (coordinate position relative to the real object A) of the virtual object OB, and the detected coordinate position (coordinate position relative to the user) of the real object A (S9), and generates display data including the virtual object OB (S10)”, Fig 22, par 0186, “Behavior control of the virtual object OB by the AR processing unit 142b will be described below with reference to FIGS. 22 to 24. As described above, the AR processing unit 142b controls the behavior of the virtual object OB on the basis of the detected coordinate position and shape of the real object A. FIG. 22 is a transition diagram illustrating a first example of the behavior control of the virtual object OB by the AR processing unit 142b. In the example illustrated in FIG. 22, the AR processing unit 142b changes the virtual object OB so that the overlapping portion OBa of the virtual object OB with the real object A disappears when (it is estimated that) the virtual object OB is located in the back of the real object A from the viewpoint of the user on the basis of the detected coordinate position of the real object A and the set coordinate position of the virtual object OB. Specifically, as illustrated in the drawing, when the virtual object OB is caused to move to the back of the wall A1 from the viewpoint of the user, the virtual object OB is changed so that the overlapping portion OBa with the wall A1 disappears from the viewpoint of the user”),
cause artwork to be displayed on a user terminal in a case where the current position in the three-dimensional space associated with the virtual moving object is included within an imaging range of a camera the user terminal (Fig 22, par 0186, “FIG. 22 is a transition diagram illustrating a first example of the behavior control of the virtual object OB by the AR processing unit 142b. In the example illustrated in FIG. 22, the AR processing unit 142b changes the virtual object OB so that the overlapping portion OBa of the virtual object OB with the real object A disappears when (it is estimated that) the virtual object OB is located in the back of the real object A from the viewpoint of the user on the basis of the detected coordinate position of the real object A and the set coordinate position of the virtual object OB. Specifically, as illustrated in the drawing, when the virtual object OB is caused to move to the back of the wall A1 from the viewpoint of the user, the virtual object OB is changed so that the overlapping portion OBa with the wall A1 disappears from the viewpoint of the user. In this way, the virtual object OB is displayed as if the virtual object is hidden by the wall A1 from the viewpoint of the user”) , while the camera of the user terminal is pointed toward a sky (Fig 22, par 0181, “As illustrated in FIG. 21, the head-mounted display 100b first captures an image of an external scene SC by the use of the camera 61 (S1). The image analyzing unit 144 recognizes a marker MK in the image captured by the camera 61 and recognizes the real object A (S2). In the example illustrated in FIG. 2, the walls A1 and A2 for recognizing the marker MK are recognized from the captured image” …captured image toward sky by HMD device).
But Sasaki keeps silent for teaching accept, from the user terminal, a posting of commentary on the artwork while the camera of the user terminal is pointed toward the sky..
PNG
media_image2.png
436
360
media_image2.png
Greyscale
Ono et al. teach accept, from the user terminal, a posting of commentary on the artwork displayed on the user terminal, while the camera of the user terminal is pointed toward the sky. (par 0006, “generates commentary, the commentary including first commentary generated based on behavior information corresponding to a behavior of a user and second commentary associated with the first commentary; and outputs the commentary to be available to other users “, par 0061-0062, “a procedure for receiving consent from users of the client apparatus 100 may be performed regarding the use of a captured image of the camera 330 as a generation material of commentary”, Fig 4, par 0091-0097, “The user object 1201 that is the shape of various vehicles in the list display screen 1100 may be arranged as a human type in a vehicle object described later. The user object 1201 in the shape of an unknown object or in the shape of a terminal left may be displayed in the air ….the vehicle object 1205 of a train and the vehicle object 1205 of a car are arranged in the virtual space 1207. In addition, a bus, a bicycle, an airplane, an elevator and the like may be displayed. The vehicle object 1205 is arranged in a position suitable to each vehicle in the virtual space 1207 ….in the virtual space display screen 1200, though not illustrated, another posting button may be displayed such that a snap shot of the virtual space 1207 including the user object 1201, the user information 1203, and the vehicle objects 1205 are acquired to post a message by the user with the snap shot as an attached image on the social media 520”, par 0126-0127, “The commentary image 2109 is an image generated by the image generation function 450. In the illustrated example, a first speaker image 2111 and a second speaker image 2113 representing an announcer and a commentator in a commentary booth respectively are drawn in the commentary image 2109 and a first balloon 2115 and a second balloon 2117 are drawn as a dialog therebetween “, par 0148, “commentary is also generated when the user posts a message or an image on social media and/or an image is captured by the client apparatus 100 and information containing an image generated based on the commentary may be posted on the social media 520”, Fig 9, par 0150-0154, “The image 2219 is, as described above, an image posted on the social media 520 by the user. In the illustrated example, the image 2219 is arranged near an intermediate position between the first speaker image 2211 and the second speaker image 2213 in the commentary image 2209 and the commentary image 2209 represents how an announcer and a commentator in the commentary booth are talking about the image 2219 “, par 0158-0161, “The behavior icon 2307 is displayed in a position where commentary is generated based on, for example, continuation of user's behavior. In the illustrated example, the behavior icon 2307 is displayed in a position where commentary is generated because the user's state of being on a train or walking for an extended predetermined time (for example, five minutes). Image data of the behavior icon 2307 in accordance with a content of commentary is stored in, for example, the image DB 460”).
It would have been obvious to a person of ordinary skill in the art at the time before the effective filing data of the claimed invention to modified Fujimaki et al. to include accept, from the user terminal, a posting of commentary on the artwork displayed on the user terminal as taught by Ono et al. to perform regarding the use of a captured image of the camera as a generation material of commentary and output the commentary to be available to other users to provide the behavior sharing service and information posted on the social media.
Claim(s) 8-9 is/are rejected under 35 U.S.C. 103 as being unpatentable over U.S. PGPubs 2016/0033770 to Fujimaki et al. in view of U.S. PGPubs 2020/0186608 to Ono et al., further in view of U.S. PGPubs 2018/0181596 to Li.
Regarding claim 8, Fujimaki et al. as modified by Ono et al. teach all the limitation of claim 1, but keep silent for teaching wherein the at least one program, when executed by the at least one processor, causes the information processing device to: receive, from the user terminal, a message indicating that the virtual moving object is included within the imaging range of the camera of the user terminal; acquire an image of the virtual moving object in response to receiving the message indicating that the virtual moving object is included within the imaging range of the camera of the user terminal; and cause the image of the virtual moving object to be displayed on the user terminal.
In related endeavor, Li teaches wherein the at least one program, when executed by the at least one processor, causes the information processing device to: receive, from the user terminal, a message indicating that the virtual moving object is included within the imaging range of the camera of the user terminal; acquire an image of the virtual moving object in response to receiving the message indicating that the virtual moving object is included within the imaging range of the camera of the user terminal; and cause the image of the virtual moving object to be displayed on the user terminal (par 0007-0009, “Through the system, it is not necessary for the user to be in the vicinity of a target moving object, but the user can remotely set up the virtual message associated to one of the moving objects. The moving object can be a person, a vehicle or any moving thing with specific image features. The remote management system provides information of a plurality of registered moving objects, and a database that records image information of the moving objects, and the virtual messages associated to the moving objects. The database also records configuration data such as a viewable range, a user range and/or a time period with respect to every virtual message. The virtual message can be a text, a picture, a video, a sound, or a combination thereof …. The server receives the data of the virtual message associated to the moving object through the remote management interface. The server also receives searching criteria corresponding to the virtual message …. Through an image processing process, image information of the image of the moving object can be obtained. In the meantime, the mobile device generates positioning information related to the moving object. The positioning information is mainly a ground-position data that renders a distance relationship between the mobile device and the moving object. …..The virtual message in the search result can be displayed on the mobile device”, par 0029-0030, “The remote management system renders a remote management interface provided for any user to register a moving object and as well set up a virtual message. A searchable database can be built, and a search engine is therefore established. The search engine allows the user to search the virtual message according to a captured moving object. he following diagram schematically shows a circumstance describing the method that a user uses a mobile device to create a virtual message onto a moving object.”, par 0033-0035, “The user manipulates the mobile device 10 to point toward the moving object 101, and places the virtual message 103 that is instantly displayed on the screen of the mobile device 10 on the moving object 101 …. When the user confirms that the virtual message 103 has been annotated onto a position of the moving object 101, the software program executed in the mobile device 10 transmits the image information of the position of the moving object 101 corresponding to the virtual message 103 to the server of the system. ….. The image information of the moving object 101 can be the color block data and/or the line data between the color blocks of the moving object 101 associated to the virtual message 103. It should be noted that the search mode can be performed by another software program executed in the mobile device 10. The color blocks extracted from the moving object 101 are formed by the cloth patterns of the clothing of the person. The positioning information of the mobile device 10, and the color block data and/or line data of the moving object 101 render the references for displaying the virtual message 103”).
It would have been obvious to a person of ordinary skill in the art at the time before the effective filing data of the claimed invention to modified Fujimaki et al. as modified by Ono et al. to include wherein the at least one program, when executed by the at least one processor, causes the information processing device to: receive, from the user terminal, a message indicating that the virtual moving object is included within the imaging range of the camera of the user terminal; acquire an image of the virtual moving object in response to receiving the message indicating that the virtual moving object is included within the imaging range of the camera of the user terminal; and cause the image of the virtual moving object to be displayed on the user terminal as taught by Li to provide an open platform for users to set up a virtual image using an augmented reality technology over a moving object to facilitate user to conduct remote management of managing multiple virtual messages and their associated moving objects at the same time.
Regarding claim 9, Fujimaki et al. as modified by Ono et al. and Li teach all the limitation of claim 8, and Li further teaches wherein the at least one program, when executed by the at least one processor, causes the information processing device to: transmit, to the user terminal, a message indicating a trajectory of the virtual moving object, wherein the message indicating that the virtual moving object is included within the imaging range of the camera of the user terminal is received after the message indicating the trajectory of the virtual moving object is transmitted (par 0011, “a remote management interface is provided, and a virtual message transmitted through the remote management interface is received over a network. The server also receives a moving object selected through the remote management interface, and a correlation between the moving object and the virtual message is therefore established. After that, positioning information for searching the virtual message over the moving object can be obtained. The positioning information is such as the image information of the moving object, and a distance relationship between the moving object and the user who searches the virtual message”, par 0034, “When the user confirms that the virtual message 103 has been annotated onto a position of the moving object 101, the software program executed in the mobile device 10 transmits the image information of the position of the moving object 101 corresponding to the virtual message 103 to the server of the system”, par 0036, “one of the parameters of the viewable range associated to the virtual message 103 is a distance between the mobile device 10 held by the user and the moving object 101. This distance between the mobile device 10 and the moving object 101 can be determined by a proximity sensor of the mobile device 10, or through the image information”, par 0054-0056, “the user can utilize the mobile device to initiate the software program and activate a camera module to capture an image of a moving object. The image information associated to the moving object can be extracted. The software program obtains the positioning information such as ground-position data and/or spatial angle data generated by a positioning circuit of the mobile device. The image information and the positioning information are transmitted to the server 301. The system provides a search result when compared with the data in the database. The search result is generated based on the image information, e.g. the color block data, line data and/or facial recognition, and the positioning information of the mobile device. After that, the virtual message in the search result is displayed on the display of the mobile device”, par 0067, “A data transmitting and receiving instruction 476 is used to establish a connection between the mobile device 400 and the server 41, and to transmit or receive signals there-between. For example, the instruction 476 is operated to transmit the positioning information and the image information, and receive the search result from the server 41. When the virtual message is searched, it will be listed in the search result. Afterwards, a virtual message display instruction 477 is operated to display the virtual message which can be overlapped with the moving object that is associated to the virtual message.”).
Claim(s) 10-11 is/are rejected under 35 U.S.C. 103 as being unpatentable over U.S. PGPubs 2016/0033770 to Fujimaki et al. in view of U.S. PGPubs 2020/0186608 to Ono et al., further in view of U.S. PGPubs 2013/0326366 to Choi et al.
Regarding claim 10, Fujimaki et al. as modified by Ono et al. teach all the limitation of claim 1, but keep silent for teaching wherein the at least one program, when executed by the at least one processor, causes the information processing device to: receive, from the user terminal, a message indicating that a zoom-in operation has been performed; and acquire the artwork in response to receiving the message indicating that the zoom-in operation has been performed.
In related endeavor, Choi et al. teach wherein the at least one program, when executed by the at least one processor, causes the information processing device to: receive, from the user terminal, a message indicating that a zoom-in operation has been performed; and acquire the artwork in response to receiving the message indicating that the zoom-in operation has been performed (par 0080-0085, “When there are a plurality of other user terminal apparatuses 200, and when the location of the user terminal apparatus 100 and the locations of all of the other user terminal apparatuses 200 are displayed on the map, and the zoom-in command is input, the location of the user terminal apparatus 100 and locations of some of the other user terminal apparatuses 200 are displayed on the map which is magnified according to the zoom-in command”, par 0109-0112, “the method may further include receiving input of a message through the user terminal apparatus (S1140), and displaying the input message on a region corresponding to the location of the user terminal apparatus on the map (S1150). Referring to FIG. 12, the method may further include, when a zoom-in command is input through the user terminal apparatus (S1240), zooming in on the map with reference to the location of the user terminal apparatus (S1245), and, when a zoom-out command is input through the user terminal apparatus (S1250), zooming out on the map with reference to the location of the user terminal apparatus (S1255) …..Referring to FIG. 14, when there are a plurality of other user terminal apparatuses, the method may further include: when locations of some of the plurality of other user terminal apparatuses are displayed on the map (S1420) and a message input through one of the plurality of other user terminal apparatuses that is not displayed on the map is received (S1425), displaying a location identifier and the message on a corresponding region on the screen based on a relative location of one of the plurality of other user terminal apparatuses through which the message is input with respect to the user terminal apparatus (S1430). When locations of all of the plurality of other user terminal apparatuses are displayed on the map (S1420), or a message input through one of the plurality of other user terminal apparatuses that is displayed on the map is received (S1425), messages are displayed on regions corresponding to the actual locations of the user terminal apparatuses, on the map (S1440)”).
It would have been obvious to a person of ordinary skill in the art at the time before the effective filing data of the claimed invention to modified Fujimaki et al. as modified by Ono et al. to include wherein the at least one program, when executed by the at least one processor, causes the information processing device to: receive, from the user terminal, a message indicating that a zoom-in operation has been performed; and acquire the artwork in response to receiving the message indicating that the zoom-in operation has been performed as taught by Choi et al. to provide location-based service information on a user's location that is necessary in real life to provide a new user experience by combining a social network and a virtual world providing real world information.
Regarding claim 11, Sasaki as modified by Ono et al. and Choi et al. teaches all the limitation of claim 10, and Choi et al. further teach wherein the at least one program, when executed by the at least one processor, causes the information processing device to: acquire at least one user input image in response to receiving the message indicating that the zoom-in operation has been performed; and cause the artwork and the at least one user input image to be displayed together on the user terminal (par 0016-0017, “when the zoom-in command is input, the controller may control to display the location of the user terminal apparatus and locations of some of the plurality of other user terminal apparatuses on the map which is magnified according to the zoom-in command, and, when the location of the user terminal apparatus and locations of some of the plurality of other user terminal apparatuses are displayed on the map, and when the zoom-out command is input, the controller may control to display the location of the user terminal apparatus and the locations of all of the plurality of other user terminal apparatuses on the map which is reduced according to the zoom-out command”, par 0079-0085, “when a zoom-in or zoom-out command is input through the user interface 140, the controller 130 of the user terminal apparatus 100 controls the display 110 to zoom in or out on the map with reference to the location of the user terminal apparatus. Zoom refers to a function of magnifying or reducing the scale of the map. That is, the zoom-in command is to magnify the map with reference to a certain point on the map displayed by the display 110 and increase the scale of the map, and the zoom-out command is to reduce the map with reference to a certain point on the map displayed by the display 110 and reduce the scale of the map“, par 0110-0111, “displaying the location of the user terminal apparatus and locations of some of the plurality of other user terminal apparatuses on the map which is magnified according to the zoom-in command (S1346), and, when the location of the user terminal apparatus and locations of some of the plurality of other user terminal apparatuses are displayed on the map (S1354), and the zoom-out command is input (S1352) in the state in which, displaying the location of the user terminal apparatus and the locations of all of the plurality of other user terminal apparatuses on the map which is reduced according to the zoom-out command (S1356)”).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Jin Ge whose telephone number is (571)272-5556. The examiner can normally be reached 8:00 to 5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jason Chan can be reached at (571)272-3022. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
JIN . GE
Examiner
Art Unit 2619
/JIN GE/Primary Examiner, Art Unit 2619
Prosecution Insights