DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
This is in response to applicant’s amendment/response filed on 04/02/2026, which has been entered and made of record. Claim 1 has been amended. Claims 8-11 have been added. Claims 1-11 are pending in the application wherein Claims 2-7 are withdrawn.
Response to Arguments
Applicant's arguments filed on 04/02/2026 have been fully considered but they are rendered moot in view of the new grounds of rejection presented below (as necessitated by the amendment to claim 1).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1 is/are rejected under 35 U.S.C. 103 as being unpatentable over U.S. PGPubs 2015/0379777 to Sasaki in view of U.S. PGPubs 2020/0186608 to Ono et al., further in view of U.S. PGPubs 2022/0415193 to Oi et al..
Regarding claim 1, Sasaki teaches an information processing device comprising: at least one processor; and at least one memory storing at least one program that, when executed by the at least one processor, causes the information processing device to (Fig 2, par 0020, par 0096-0098):
acquire information indicating a current position of a virtual moving object moving aboveground over time (par 0010, “a position identification element configured to, after decision by the decision element that the sensor is located at the reference position, identify a current position of the sensor based on the reference position stored in the first storage element and the information on the movement measured by the sensor “, par 0121-122, “the candidate information 112 is graphic information of a virtual object displayed in augmented reality, information on the position thereof, information on time thereof, and map information in the augmented reality space 9 (i.e., layout data) “, par 0127-0129, “thereby identifying the current position of the portable terminal device 2 (group of sensors 24) and identifying the current posture of the portable terminal device (group of sensors 24). The absolute position of the starting point of the moving route is the reference position contained in the reference information 103 “, par 0145-0147, “When the field of view in the augmented reality space 9 is determined, a thing to be virtually displayed (virtual object) corresponding to that field of view and the shape of that thing are determined, for example. In this manner, the augmented reality formation unit 202 can select appropriate output information 215 from the candidate information 112. To the principle for selecting the output information 215 based on the position information 214 after creation of that position information 214, a conventional technique can be applied as appropriate ….the virtual object to be displayed (e.g., the guide route) has to be changed when the user's destination is different even if the current position of the user (the position information 214) is the same. That is, the output information 215 has to be selected in accordance with information different from the position information 214, such as the destination”),
cause artwork to be displayed on a user terminal in a case where the current position of the virtual moving object is included within an imaging range of a camera the user terminal (Figs 5-6, par 0153-0167, “The portable terminal device 2 can create and provide the augmented reality display screen by superimposing the view of the augmented environment formed by the virtual objects (the route 215a and the advertisements 215b, 215c, and 215d) on the view of the real environment (the image in shop 213a), as shown in FIG. 5 …. the augmented reality formation unit 202 can decide information related to a shop the user does not want to go from the owner information 211, and can prevent the advertisements 215b, 215c, and 215d of shops other than the shop D from being displayed …. FIG. 6 shows that an image in shop 215e and a route 215f, a start mark 215g, and coupons 215h and 215i that are virtual objects are displayed on the screen of the liquid crystal display. The image in shop 215e is not captured image information 213 captured by the image capturing unit 25, but is a map image obtained by deforming the inside of the actual shop, that is, an image representing the real portion in the augmented reality space 9 …. The route 215f is information that is calculated by the augmented reality formation unit 202 based on the position information 214, the owner information 211, and the candidate information 112 (map information), and is represented by the output information 215 (diagram) selected from the candidate information 112. The route 215f is an image representing the augmented portion (virtual portion) in the augmented reality space 9”, Figs 21-22, par 0550-0253, “As illustrated in FIG. 21, the captured image information 213d as the real portion, a treasure box image 215s as the target, and a message 215t and a compass image 215u that indicate information as a clue for the search are synthesized and displayed on the display unit 23. The treasure box image 215s, the message 215t, and the compass image 215u are the output information 215 selected from the candidate information 112”).
But Sasaki keeps silent for teaching accept, from the user terminal, a posting of commentary on the artwork while the camera of the user terminal is pointed toward the sky..
PNG
media_image1.png
436
360
media_image1.png
Greyscale
Ono et al. teach accept, from the user terminal, a posting of commentary on the artwork displayed on the user terminal, while the camera of the user terminal is pointed toward the sky. (par 0006, “generates commentary, the commentary including first commentary generated based on behavior information corresponding to a behavior of a user and second commentary associated with the first commentary; and outputs the commentary to be available to other users “, par 0061-0062, “a procedure for receiving consent from users of the client apparatus 100 may be performed regarding the use of a captured image of the camera 330 as a generation material of commentary”, Fig 4, par 0091-0097, “The user object 1201 that is the shape of various vehicles in the list display screen 1100 may be arranged as a human type in a vehicle object described later. The user object 1201 in the shape of an unknown object or in the shape of a terminal left may be displayed in the air ….the vehicle object 1205 of a train and the vehicle object 1205 of a car are arranged in the virtual space 1207. In addition, a bus, a bicycle, an airplane, an elevator and the like may be displayed. The vehicle object 1205 is arranged in a position suitable to each vehicle in the virtual space 1207 ….in the virtual space display screen 1200, though not illustrated, another posting button may be displayed such that a snap shot of the virtual space 1207 including the user object 1201, the user information 1203, and the vehicle objects 1205 are acquired to post a message by the user with the snap shot as an attached image on the social media 520”, par 0126-0127, “The commentary image 2109 is an image generated by the image generation function 450. In the illustrated example, a first speaker image 2111 and a second speaker image 2113 representing an announcer and a commentator in a commentary booth respectively are drawn in the commentary image 2109 and a first balloon 2115 and a second balloon 2117 are drawn as a dialog therebetween “, par 0148, “commentary is also generated when the user posts a message or an image on social media and/or an image is captured by the client apparatus 100 and information containing an image generated based on the commentary may be posted on the social media 520”, Fig 9, par 0150-0154, “The image 2219 is, as described above, an image posted on the social media 520 by the user. In the illustrated example, the image 2219 is arranged near an intermediate position between the first speaker image 2211 and the second speaker image 2213 in the commentary image 2209 and the commentary image 2209 represents how an announcer and a commentator in the commentary booth are talking about the image 2219 “, par 0158-0161, “The behavior icon 2307 is displayed in a position where commentary is generated based on, for example, continuation of user's behavior. In the illustrated example, the behavior icon 2307 is displayed in a position where commentary is generated because the user's state of being on a train or walking for an extended predetermined time (for example, five minutes). Image data of the behavior icon 2307 in accordance with a content of commentary is stored in, for example, the image DB 460”).
It would have been obvious to a person of ordinary skill in the art at the time before the effective filing data of the claimed invention to modified Sasaki to include accept, from the user terminal, a posting of commentary on the artwork displayed on the user terminal as taught by Ono et al. to perform regarding the use of a captured image of the camera as a generation material of commentary and output the commentary to be available to other users to provide the behavior sharing service and information posted on the social media.
But Sasaki as modified by Ono et al. keep silent for teaching cause artwork to be displayed on a user terminal in a case where the current position of the virtual moving object is included within an imaging range of a camera of the user terminal, while the camera of the user terminal is pointed toward a sky.
PNG
media_image2.png
328
450
media_image2.png
Greyscale
In related endeavor, Oi et al. teach cause artwork to be displayed on a user terminal in a case where the current position of the virtual moving object is included within an imaging range of a camera of the user terminal, while the camera of the user terminal is pointed toward a sky (Fig 1, par 0081-0084, “processing in which the user captures an image of the drone in the sky by using a camera of a camera-equipped user terminal such as a smartphone, and the captured image is displayed on a display unit while a flight path and scheduled flight path of the drone is being superimposed and displayed on the image …. The user 1 directs the camera of the user terminal 10 toward the drone 20 in the sky and captures an image of the drone 20”, Fig 2, par 0085-0092, “The image illustrated in FIG. 2 is an image captured by the camera of the user terminal 10, and the drone 20 is displayed therein. The image of the drone 20 is a real object actually flying in the sky. Lines indicating a flight path and scheduled flight path of the drone 20 are displayed on this real image as virtual images generated by a data processing unit of the user terminal 10. That is, an augmented reality (AR) image in which a virtual line indicating the flight path is superimposed on the real image of the drone is generated and displayed.”, Fig 7, par 0122, “if a position on the flight path indicated in the NED coordinate system can be converted into a position indicated in the camera coordinate system, the path can be accurately output to the display image shown in the camera coordinate system”).
It would have been obvious to a person of ordinary skill in the art at the time before the effective filing data of the claimed invention to modified Sasaki as modified by Ono et al. to include artwork to be displayed on a user terminal in a case where the current position of the virtual moving object is included within an imaging range of a camera of the user terminal, while the camera of the user terminal is pointed toward a sky as taught by Oi et al. to realize a configuration capable of accurately displaying a moving path of a moving object on an actually captured image of the moving object in AR display using a user terminal.
Claim(s) 8-9 is/are rejected under 35 U.S.C. 103 as being unpatentable over U.S. PGPubs 2015/0379777 to Sasaki in view of U.S. PGPubs 2020/0186608 to Ono et al., further in view of U.S. PGPubs 2022/0415193 to Oi et al., further in view of U.S. PGPubs 2018/0181596 to Li.
Regarding claim 8, Sasaki as modified by Ono et al. and Oi et al. teaches all the limitation of claim 1, but keeps silent for teaching wherein the at least one program, when executed by the at least one processor, causes the information processing device to: receive, from the user terminal, a message indicating that the virtual moving object is included within the imaging range of the camera of the user terminal; acquire an image of the virtual moving object in response to receiving the message indicating that the virtual moving object is included within the imaging range of the camera of the user terminal; and cause the image of the virtual moving object to be displayed on the user terminal.
In related endeavor, Li teaches wherein the at least one program, when executed by the at least one processor, causes the information processing device to: receive, from the user terminal, a message indicating that the virtual moving object is included within the imaging range of the camera of the user terminal; acquire an image of the virtual moving object in response to receiving the message indicating that the virtual moving object is included within the imaging range of the camera of the user terminal; and cause the image of the virtual moving object to be displayed on the user terminal (par 0007-0009, “Through the system, it is not necessary for the user to be in the vicinity of a target moving object, but the user can remotely set up the virtual message associated to one of the moving objects. The moving object can be a person, a vehicle or any moving thing with specific image features. The remote management system provides information of a plurality of registered moving objects, and a database that records image information of the moving objects, and the virtual messages associated to the moving objects. The database also records configuration data such as a viewable range, a user range and/or a time period with respect to every virtual message. The virtual message can be a text, a picture, a video, a sound, or a combination thereof …. The server receives the data of the virtual message associated to the moving object through the remote management interface. The server also receives searching criteria corresponding to the virtual message …. Through an image processing process, image information of the image of the moving object can be obtained. In the meantime, the mobile device generates positioning information related to the moving object. The positioning information is mainly a ground-position data that renders a distance relationship between the mobile device and the moving object. …..The virtual message in the search result can be displayed on the mobile device”, par 0029-0030, “The remote management system renders a remote management interface provided for any user to register a moving object and as well set up a virtual message. A searchable database can be built, and a search engine is therefore established. The search engine allows the user to search the virtual message according to a captured moving object. he following diagram schematically shows a circumstance describing the method that a user uses a mobile device to create a virtual message onto a moving object.”, par 0033-0035, “The user manipulates the mobile device 10 to point toward the moving object 101, and places the virtual message 103 that is instantly displayed on the screen of the mobile device 10 on the moving object 101 …. When the user confirms that the virtual message 103 has been annotated onto a position of the moving object 101, the software program executed in the mobile device 10 transmits the image information of the position of the moving object 101 corresponding to the virtual message 103 to the server of the system. ….. The image information of the moving object 101 can be the color block data and/or the line data between the color blocks of the moving object 101 associated to the virtual message 103. It should be noted that the search mode can be performed by another software program executed in the mobile device 10. The color blocks extracted from the moving object 101 are formed by the cloth patterns of the clothing of the person. The positioning information of the mobile device 10, and the color block data and/or line data of the moving object 101 render the references for displaying the virtual message 103”).
It would have been obvious to a person of ordinary skill in the art at the time before the effective filing data of the claimed invention to modified Sasaki as modified by Ono et al. and Oi et al. to include wherein the at least one program, when executed by the at least one processor, causes the information processing device to: receive, from the user terminal, a message indicating that the virtual moving object is included within the imaging range of the camera of the user terminal; acquire an image of the virtual moving object in response to receiving the message indicating that the virtual moving object is included within the imaging range of the camera of the user terminal; and cause the image of the virtual moving object to be displayed on the user terminal as taught by Li to provide an open platform for users to set up a virtual image using an augmented reality technology over a moving object to facilitate user to conduct remote management of managing multiple virtual messages and their associated moving objects at the same time.
Regarding claim 9, Sasaki as modified by Ono et al., Oi et al., and Li teaches all the limitation of claim 8, and further teaches wherein the at least one program, when executed by the at least one processor, causes the information processing device to: transmit, to the user terminal, a message indicating a trajectory of the virtual moving object, wherein the message indicating that the virtual moving object is included within the imaging range of the camera of the user terminal is received after the message indicating the trajectory of the virtual moving object is transmitted (Oi: Fig 2, par 0085-0093, “The image illustrated in FIG. 2 is an image captured by the camera of the user terminal 10, and the drone 20 is displayed therein. The image of the drone 20 is a real object actually flying in the sky. Lines indicating a flight path and scheduled flight path of the drone 20 are displayed on this real image as virtual images generated by a data processing unit of the user terminal 10. That is, an augmented reality (AR) image in which a virtual line indicating the flight path is superimposed on the real image of the drone is generated and displayed …. The user terminal can receive information regarding the flight path and scheduled flight path of the drone from the drone or a control center. As illustrated in FIG. 3, the user terminal 10 can receive flight path information that is the information regarding the flight path and scheduled flight path of the drone from the drone 20 or a drone management server 30 that is the control center.“, Li: par 0011, “a remote management interface is provided, and a virtual message transmitted through the remote management interface is received over a network. The server also receives a moving object selected through the remote management interface, and a correlation between the moving object and the virtual message is therefore established. After that, positioning information for searching the virtual message over the moving object can be obtained. The positioning information is such as the image information of the moving object, and a distance relationship between the moving object and the user who searches the virtual message”, par 0034, “When the user confirms that the virtual message 103 has been annotated onto a position of the moving object 101, the software program executed in the mobile device 10 transmits the image information of the position of the moving object 101 corresponding to the virtual message 103 to the server of the system”, par 0036, “one of the parameters of the viewable range associated to the virtual message 103 is a distance between the mobile device 10 held by the user and the moving object 101. This distance between the mobile device 10 and the moving object 101 can be determined by a proximity sensor of the mobile device 10, or through the image information”, par 0054-0056, “the user can utilize the mobile device to initiate the software program and activate a camera module to capture an image of a moving object. The image information associated to the moving object can be extracted. The software program obtains the positioning information such as ground-position data and/or spatial angle data generated by a positioning circuit of the mobile device. The image information and the positioning information are transmitted to the server 301. The system provides a search result when compared with the data in the database. The search result is generated based on the image information, e.g. the color block data, line data and/or facial recognition, and the positioning information of the mobile device. After that, the virtual message in the search result is displayed on the display of the mobile device”, par 0067, “A data transmitting and receiving instruction 476 is used to establish a connection between the mobile device 400 and the server 41, and to transmit or receive signals there-between. For example, the instruction 476 is operated to transmit the positioning information and the image information, and receive the search result from the server 41. When the virtual message is searched, it will be listed in the search result. Afterwards, a virtual message display instruction 477 is operated to display the virtual message which can be overlapped with the moving object that is associated to the virtual message.”).
Claim(s) 10-11 is/are rejected under 35 U.S.C. 103 as being unpatentable over U.S. PGPubs 2015/0379777 to Sasaki in view of U.S. PGPubs 2020/0186608 to Ono et al., further in view of U.S. PGPubs 2022/0415193 to Oi et al., further in view of U.S. PGPubs 2013/0326366 to Choi et al.
Regarding claim 10, Sasaki as modified by Ono et al. and Oi et al. teaches all the limitation of claim 1, but keeps silent for teaching wherein the at least one program, when executed by the at least one processor, causes the information processing device to: receive, from the user terminal, a message indicating that a zoom-in operation has been performed; and acquire the artwork in response to receiving the message indicating that the zoom-in operation has been performed.
In related endeavor, Choi et al. teach wherein the at least one program, when executed by the at least one processor, causes the information processing device to: receive, from the user terminal, a message indicating that a zoom-in operation has been performed; and acquire the artwork in response to receiving the message indicating that the zoom-in operation has been performed (par 0080-0085, “When there are a plurality of other user terminal apparatuses 200, and when the location of the user terminal apparatus 100 and the locations of all of the other user terminal apparatuses 200 are displayed on the map, and the zoom-in command is input, the location of the user terminal apparatus 100 and locations of some of the other user terminal apparatuses 200 are displayed on the map which is magnified according to the zoom-in command”, par 0109-0112, “the method may further include receiving input of a message through the user terminal apparatus (S1140), and displaying the input message on a region corresponding to the location of the user terminal apparatus on the map (S1150). Referring to FIG. 12, the method may further include, when a zoom-in command is input through the user terminal apparatus (S1240), zooming in on the map with reference to the location of the user terminal apparatus (S1245), and, when a zoom-out command is input through the user terminal apparatus (S1250), zooming out on the map with reference to the location of the user terminal apparatus (S1255) …..Referring to FIG. 14, when there are a plurality of other user terminal apparatuses, the method may further include: when locations of some of the plurality of other user terminal apparatuses are displayed on the map (S1420) and a message input through one of the plurality of other user terminal apparatuses that is not displayed on the map is received (S1425), displaying a location identifier and the message on a corresponding region on the screen based on a relative location of one of the plurality of other user terminal apparatuses through which the message is input with respect to the user terminal apparatus (S1430). When locations of all of the plurality of other user terminal apparatuses are displayed on the map (S1420), or a message input through one of the plurality of other user terminal apparatuses that is displayed on the map is received (S1425), messages are displayed on regions corresponding to the actual locations of the user terminal apparatuses, on the map (S1440)”).
It would have been obvious to a person of ordinary skill in the art at the time before the effective filing data of the claimed invention to modified Sasaki as modified by Ono et al. and Oi et al. to include wherein the at least one program, when executed by the at least one processor, causes the information processing device to: receive, from the user terminal, a message indicating that a zoom-in operation has been performed; and acquire the artwork in response to receiving the message indicating that the zoom-in operation has been performed as taught by Choi et al. to provide location-based service information on a user's location that is necessary in real life to provide a new user experience by combining a social network and a virtual world providing real world information.
Regarding claim 11, Sasaki as modified by Ono et al., Oi et al., and Choi et al. teaches all the limitation of claim 10, and Choi et al. further teach wherein the at least one program, when executed by the at least one processor, causes the information processing device to: acquire at least one user input image in response to receiving the message indicating that the zoom-in operation has been performed; and cause the artwork and the at least one user input image to be displayed together on the user terminal (par 0016-0017, “when the zoom-in command is input, the controller may control to display the location of the user terminal apparatus and locations of some of the plurality of other user terminal apparatuses on the map which is magnified according to the zoom-in command, and, when the location of the user terminal apparatus and locations of some of the plurality of other user terminal apparatuses are displayed on the map, and when the zoom-out command is input, the controller may control to display the location of the user terminal apparatus and the locations of all of the plurality of other user terminal apparatuses on the map which is reduced according to the zoom-out command”, par 0079-0085, “when a zoom-in or zoom-out command is input through the user interface 140, the controller 130 of the user terminal apparatus 100 controls the display 110 to zoom in or out on the map with reference to the location of the user terminal apparatus. Zoom refers to a function of magnifying or reducing the scale of the map. That is, the zoom-in command is to magnify the map with reference to a certain point on the map displayed by the display 110 and increase the scale of the map, and the zoom-out command is to reduce the map with reference to a certain point on the map displayed by the display 110 and reduce the scale of the map“, par 0110-0111, “displaying the location of the user terminal apparatus and locations of some of the plurality of other user terminal apparatuses on the map which is magnified according to the zoom-in command (S1346), and, when the location of the user terminal apparatus and locations of some of the plurality of other user terminal apparatuses are displayed on the map (S1354), and the zoom-out command is input (S1352) in the state in which, displaying the location of the user terminal apparatus and the locations of all of the plurality of other user terminal apparatuses on the map which is reduced according to the zoom-out command (S1356)”).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Jin Ge whose telephone number is (571)272-5556. The examiner can normally be reached 8:00 to 5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jason Chan can be reached at (571)272-3022. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
JIN . GE
Examiner
Art Unit 2619
/JIN GE/ Primary Examiner, Art Unit 2619
Prosecution Insights