Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 2, 7, and 8 are rejected under 35 U.S.C. 103 as being unpatentable over “Meta Cookie+: An Illusion-Based Gustatory Display” by Takuji Narumi, et al. (hereinafter Narumi) in view of U.S. Patent Application Publication 2020/0312033 A1 (hereinafter Ohashi).
Regarding claim 1, the limitations “An information processing apparatus comprising at least one processor, the processor carrying out: a detection process of detecting a predetermined detection target in an image that is obtained by capturing at least a part of a field of view of a user and shows a food item; an overlay area setting process of setting, in the image, an overlay area for an overlay image including information related to the food item by using the detection target as a reference; an overlay image generating process of generating the overlay image; and a display control process of performing overlay display of the information on the overlay area, wherein the display control process performs overlay display of an image of a predetermined food in the overlay area” are taught by Narumi (Narumi, e.g. abstract, sections 1-5, describes the meta cookie system for pseudo-gustation, which is an augmented reality system comprising a head mounted display having cameras observing the user’s field of view to detect cookies with edible markers in order to overlay images of different selected cookies on the detected cookie/marker for display to the user, e.g. section 3. Narumi, e.g. sections 3.1-3.3, describes the edible marker system which estimates the 6DOF coordinates, occlusion, and division of a marker printed onto the cookie by detecting the marker and estimating a homography matrix using a template image of the marker and the detected feature points of the marker, e.g. Figure 3, step 1, corresponding to the claimed detection process of detecting a predetermined detection target in an image, obtained by capturing part of the user’s field of view, which contains a food item. Further, Narumi, e.g. sections 3.1, 3.2, 3.4, figure 3, teaches that the location(s) of the detected marker (or marker portions) are used to subtract the background which does not contain the marker in step 2, followed by superimposing the image of the cookie flavor selected by the user, e.g. figure 1, in the non-background regions corresponding to the detected marker portion(s) in step 3, with exemplary results shown on the right of figure 3, corresponding to the claimed overlay setting process for setting an overlay area for the food item using the detection target as a reference, overlay image generating process of generating the overlay image, and the display control process of performing overlay display of the overlay image of a pre-determined food on the overlay area, i.e. the information being overlaid is an overlay image compositing the cookie image over the captured cookie, where the cookie image is the pre-determined image of the cookie flavor selected by the user from the 5 presented options as in figure 1.)
The limitations “wherein the overlay image generating process performs to: determine a food area and a background area from the image capturing the food item, process the image of the predetermined food so as to conform to the shape and size of the determined food area, composite the processed image of the predetermined food on a portion corresponding to the food area” are taught by Narumi (As noted above, Narumi, e.g. sections 3.1, 3.2, 3.4, figure 3, teaches that the location(s) of the detected marker (or marker portions) are used to subtract the background which does not contain the marker in step 2, followed by superimposing the image of the cookie flavor selected by the user, e.g. figure 1, in the non-background regions corresponding to the detected marker portion(s) in step 3. That is, as claimed, the food and background areas are determined, the image of the predetermined food is processed to conform to the shape and size of the food area, and portions of the processed food image are composited onto the corresponding portions of the detected food areas, as in the examples on the right of figure 3.)
The limitation “make the background area transparent of translucent so as to generate the overlay image” is implicitly taught by Narumi (As noted above, Narumi teaches the claimed steps of determining the food area(s), processing the predetermined food image to conform to the size and shape of the food area, and composite the portions of the processed food image onto the corresponding food area(s), as in the examples on the right of figure 3. While not explicitly stated by Narumi, one of ordinary skill in the art would have found it implicit that the background subtraction step corresponds to the claimed step of making the background area transparent or translucent to generate the overlay image, i.e. as in the example of figure 3, step 1 shows the captured image, step 2 shows the result of background subtraction excluding the background area of the captured image in the overlay area, and step 3 shows that the predetermined food image and the background subtraction result are combined to composite the cookie image onto the captured image only within the food area/overlay area. That is, the background subtraction result identifies which pixels belong to the background and should not be composited onto the captured image, effectively making them transparent or translucent, indicating that one of ordinary skill in the art would have found it implicit that Narumi’s background subtraction step corresponds to the claimed step of making the background area transparent or translucent. In the interest of compact prosecution, because Narumi does not explicitly describe the background subtraction in the claimed terms and in consideration of the possibility that Applicant could potentially show the claim limitation is not inherent to Narumi’s background subtraction step used for a compositing operation, Ohashi is cited for explicitly teaching this limitation as a chroma key type compositing operation in an analogous head mounted display system.) However, this limitation is taught by Ohashi (Ohashi, e.g. abstract, paragraphs 26-129, describes a system for superimposing computer graphics images on captured real space images using chroma key processing. Ohashi, e.g. paragraphs 33-39, 59-62, figures 3-6, explains that the superimposition involves capturing the real space image, e.g. figure 3, which can be superimposed with a rendered virtual object, e.g. figure 4, where the virtual object is rendered by using depth to determine which rendered image areas correspond to the background that is not occluded by the virtual object, which areas correspond to the virtual object that are not occluded by captured objects, and which areas correspond to captured objects which occlude the virtual object, and setting background areas and occluding captured object areas to a chroma key color that is displayed as a transparent region, i.e. as in figure 6, the background and hand are set to the black color indicating transparent regions, which when composited with the captured image, results in the combined image of figure 5 superimposing the teapot such that it appears in front of the background and behind the hand. More specifically, Ohashi, e.g. paragraphs 38, 39, indicates the background region is filled with a the specific chroma key color which is displayed as transparent in order to allow the camera image to be displayed in the chroma key region, indicating an analogous compositing operation to Narumi’s compositing operation, i.e. the result of Narumi’s background subtraction in step 2 identifies pixels where the captured image should be displayed in the combined image, which could be performed in the same manner disclosed in more detail by Ohashi, by setting the background subtraction result pixels to a chroma key color indicating a transparent region where the camera image should be displayed in the combined resulting image.)
Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to implement Narumi’s meta cookie system using Ohashi’s chroma key type compositing operation to combine the cookie image with the captured image because, as discussed above, one of ordinary skill in the art would have found it implicit that Narumi’s background subtraction step corresponds to the claimed step of making the background area transparent or translucent, and although Narumi does not explicitly indicate that the superimposition step uses the result of the background subtraction step to set transparent regions where the captured image is displayed in the combined resulting image, Ohashi, describing an analogous augmented reality compositing operation using chroma key type compositing explicitly teaches that the background region is subtracted from the rendered virtual/overlay image by setting the background region to a chroma key color which is displayed as transparent so that the captured/camera image is displayed in the chroma key region.
Regarding claim 2, the limitations “wherein the food area includes multiple food areas, and wherein the overlay image generating process performs to: process the image of the predetermined food so as to conform to the shape and size of each of the determined multiple food areas, and composite the processed image on portions corresponding to the multiple food areas so as to generate the overlay image” are taught by Narumi (As discussed in the claim 1 rejection above, Narumi, e.g. sections 3.1, 3.2, 3.4, figure 3, teaches that the location(s) of the detected marker (or marker portions) are used to subtract the background which does not contain the marker in step 2, followed by superimposing the image of the cookie flavor selected by the user, e.g. figure 1, in the non-background regions corresponding to the detected marker portion(s) in step 3. That is, portions of the processed food image are composited onto the corresponding portions of the detected food areas, as in the divided cookie example of figure 3, corresponding to the claim 2 requirement that there are multiple food areas and the processing is performed for each of the multiple food areas, i.e. the in the divided cookie example each portion of the cookie has the corresponding portion of the cookie image superimposed thereon with the proper shape and size.)
Regarding claim 7, the limitations are similar to those treated in the above rejection(s) and are met by the references as discussed in claim 1 above.
Regarding claim 8, the limitations are similar to those treated in the above rejection(s) and are met by the references as discussed in claim 1 above, except for the limitation “A non-transitory storage medium storing a display control program causing a computer to carry out” the same processes recited in claim 1. While Narumi, e.g. figure 2, indicates that a PC is used to perform the processing of Narumi’s meta cookie system, Narumi does not describe the PC, per se, or explicitly indicate the use of a non-transitory medium for storing the program controlling the meta cookie system. However, it is noted that one of ordinary skill in the art would have recognized this as an implicit, if not inherent, feature of Narumi’s system, i.e. it is conventional to store PC programs using non-transitory media such as hard disks or flash memory, such that even if it were shown that Narumi’s system did store the meta cookie control program on a transitory media, one of ordinary skill in the art would have found it obvious before the effective filing date of the claimed invention to implement Narumi’s meta cookie system, using Ohashi’s chroma key type compositing operation, using a control program stored on non-transitory media because it is the conventional way to store PC programs.
Claim 3 is rejected under 35 U.S.C. 103 as being unpatentable over “Meta Cookie+: An Illusion-Based Gustatory Display” by Takuji Narumi, et al. (hereinafter Narumi) in view of U.S. Patent Application Publication 2020/0312033 A1 (hereinafter Ohashi) as applied to claim 2 above, and further in view of U.S. Patent Application Publication 2021/0042487 A1 (hereinafter Altamirano) in view of “AR food changer using deep learning and cross-modal effects” by Junya Ueda, et al. (hereinafter Ueda).
Regarding claim 3, the limitation “the processor carrying out a correction process of subjecting the image to color correction by means of a color correction marker detected in the image, the color correction marker serving as a reference of the color correction” is not explicitly taught by Narumi (Narumi does not address performing color correction on the captured image, either generally or specifically using a color correction marker detected in the captured image, as claimed.) However this limitation is taught by Altamirano (Altamirano, e.g. abstract, paragraphs 44-101, describes a system for performing color correction on captured images by detecting a barcode marker within the captured image having an embedded color palette for performing color correction of the captured image. More specifically, Altamirano, e.g. paragraphs 46-67, describes embedding color palettes in barcode markers, where the objects in the image, which may be food as in the example of figure 4, have different apparent color values as in paragraphs 78-84, and, e.g. paragraphs 85-99, the colors of the image/objects therein are corrected based on the captured colors of the embedded palette. Finally, it is noted that one of ordinary skill in the art would recognize the general benefit of performing color correction, i.e. as in Altamirano, paragraphs 4-6, accounting for the deviations between the actual color values and the captured color values increases the accuracy of the corrected image.)
Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Narumi’s meta cookie system, using Ohashi’s chroma key type compositing operation, to include Altamirano’s color correction using color palette barcode markers in order to increase the accuracy of the captured image(s) by performing color correction. In Narumi's modified meta cookie system, Altamirano’s color palette barcode marker(s) would be placed in the scene in location(s) within the field of view of at least Narumi’s camera 1 capturing the video see through video images, and Narumi’s PC would perform Altamirano's color correction technique to correct the captured image colors. It is noted that Altamirano’s color correction technique could be applied to correct the captured image colors either before or after the 3 steps of Narumi section 3.2.
The limitation “wherein the display control process performs the overlay display of the image of the predetermined food item on an area with a predetermined color range in the overlay area undergone the color correction” is partially taught by Narumi in view of Altamirano (As noted above, in Narumi’s modified system, Altamirano’s color correction technique could be applied to correct the captured image colors either before or after the 3 steps of Narumi section 3.2, where performing the correction technique before Narumi’s 3 steps would cause Narumi’s overlay to be performed on an area in the image having already undergone the color correction, as claimed. Narumi, e.g. section 3.2, indicates that the edible markers are detected using feature points, i.e. Narumi does not teach that the overlay is performed on an area with a predetermined color range.) However, this limitation is taught by Ueda (Ueda, e.g. abstract, sections I-IV, describes an augmented reality food changer system analogous to Narumi, e.g. section II A discusses the meta cookie system as similar, with Ueda’s system improving on meta cookie by modifying the visual texture of food without placing marker(s) on the food. Ueda, section II, figure 1, describes operation of the system including capturing an image, detecting a food areas and food area contours using a combination of machine learning and predetermined color ranges associated with the target food item to be replaced, e.g. equations 3, 4, and tracking the food areas over time with occlusion support, e.g. sections II D-F. Ueda, section II G, teaches that upon recovering the rectangle representing the food area in the image, e.g. figure 8, overlays the texture image of a different food by alpha blending. That is, Ueda sets an overlay area based on detecting the area of a food item having a predetermined color range, corresponding to the claimed overlay display of the image of the predetermined food item on an area with a predetermined color range in the overlay area. It is additionally noted that one of ordinary skill in the art would recognize that Narumi’s meta cookie system could be beneficially modified to perform both types of detection/overlay, i.e. Narumi’s detection/overlay processing would be used to detect/overlay images for cookies having edible markers thereon while Ueda’s detection/overlay processing would be used to detect foods which can be recognized using machine learning and predetermined color ranges, thereby allowing the system to support multiple types of food item detection/overlay.)
Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Narumi’s meta cookie system, using Ohashi’s chroma key type compositing operation, including Altamirano’s color correction using color palette barcode markers, to perform Ueda’s markerless food item detection/overlay technique in addition to Narumi’s edible marker food item detection/overlay technique in order to allow the system to support multiple types of food item detection/overlay, i.e. as noted above one of ordinary skill in the art would recognize this to be a beneficial modification allowing both types of detection/overlay to be performed with a single system. As noted above, in Narumi’s modified system, Altamirano’s color correction technique could be applied to correct the captured image colors either before or after the 3 steps of Narumi section 3.2, where performing the correction technique before Narumi’s 3 steps would cause Narumi’s overlay to be performed on an area in the image having already undergone the color correction, as claimed. Further, in Narumi’s modified system performing Altamirano’s color correction technique prior to food item detection/overlay, and performing both types of detection/overlay, Ueda’s food item detection/overlay would set an overlay area based on detecting the area of a food item having a predetermined color range as noted above, corresponding to the claim 3 overlay processing limitation as a whole, i.e. the overlay area would be set based on detecting the area of a food item having a predetermined color range in the overlay area having already undergone color correction.
Claims 4-6 is rejected under 35 U.S.C. 103 as being unpatentable over “Meta Cookie+: An Illusion-Based Gustatory Display” by Takuji Narumi, et al. (hereinafter Narumi) in view of U.S. Patent Application Publication 2020/0312033 A1 (hereinafter Ohashi) as applied to claim 2 above, and further in view of “Transfork: Using Olfactor Device for Augmented Tasting Experience with Video See-Through Head-mounted Display” by Ying-Li Lin, et al. (hereinafter Lin) in view of U.S. Patent 8,542,906 B1 (hereinafter Persson).
Regarding claim 4, the limitation “wherein the detection target is a hand of the user or tableware item held by the user, and the overlay area setting process sets the overlay area by using the hand of the user or the tableware item held by the user as a reference” is not taught by Narumi (As discussed in the claim 1 rejection above, Narumi uses the edible marker printed on the cookie as the detection target, rather than a user’s hand or utensil.) However, this limitation is taught by Lin in view of Persson (Lin, e.g. abstract, sections 1-4, describes the TransFork system, which uses a visual see through HMD to change the color of a food item on the transfork utensil, along with an olfactory device to provide an odor, in order to simulate a different flavor of the food item, analogous to Narumi’s meta cookie system. Lin, e.g. section 2, figure 3, teaches that the transfork is tracked using a QR code on the fork, and the position of the QR code is used to change the color of the food on the fork, i.e. as shown in figure 1b, the real food item is green, but is changed to a red color in the display of the HMD. While Lin teaches the claim limitation, i.e. Lin’s detection target is a fork, a tableware item held by the user, and the overlay/augmentation area is set using the QR code on the tableware item as a reference, it is noted that Lin describes the system at a high level without significant implementation detail, and therefore in the interest of compact prosecution, Persson is cited for teaching details of displaying an AR overlay at a predetermined offset position relative to a detected AR marker. Persson, e.g. abstract, cols 1-15, describes an augmented reality system for presenting virtual images corresponding to detected AR markers, where the virtual image and relative offset are predetermined for each AR marker. More specifically, Persson, e.g. col 1, lines 20-41, col 2, line 57 – col 3, line 12, col 4, lines 15-35, teaches that the AR markers have patterns which are captured by the AR device camera and recognized by the AR device, where, e.g. col 4, lines 4-13, col 7, lines 54-66, a database stores the association of markers with corresponding virtual images and relative offsets, which, e.g. col 4, line 39 – col 5, line 60, col 7, line 40 – col 8, line 11, are used by the AR device to overlay the virtual image at the offset position relative to the detected location of the marker. Further, Persson, e.g. col 5, lines 32-60, indicates that the virtual image relative offset is a 6DOF relative offset, i.e. as in the second and third examples discussed with respect to figure 2, the virtual image is presented at a relative offset that is parallel with the detected marker and the AR device display screen, whereas in the examples discussed with respect to figure 3, the detected markers are not parallel to the AR device display screen, but the virtual images displayed at the relative offset are parallel to the AR device display screen, i.e. the relative pose is not only in 3D space, but may or may not maintain a fixed rotational relationship to the marker, indicating a 6DOF relative offset used for specifying the overlay area of the virtual image.)
Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Narumi’s meta cookie system, using Ohashi’s chroma key type compositing operation, to include Lin’s transfork food item flavor simulation technique, using Persson’s AR markers with virtual image offsets, in addition to Narumi’s edible marker cookie, in order to support flavor simulation for food items which support edible markers and food items which do not support edible markers, i.e. Lin’s exemplary food item is a jujube which may be incompatible an edible marker, i.e. even if the edible marker could be printed, it would be challenging for the user to hold it without occluding most of the marker, whereas the transfork system would be compatible with many food items that can be eaten with a fork. In Narumi’s modified system, when the detected marker is Lin’s transfork QR code rather than the edible marker, the overlay area would be specified as a relative offset from the detected marker, as taught by Persson, thereby implementing Lin’s described technique of changing the color of the food item speared by the fork using the attached QR code, corresponding to the claim 4 limitations requiring that the detection target is a tableware item held by the user, and the overlay area setting process sets the overlay using the tableware item as a reference. It is further noted that this combination also teaches the limitations of claim 5, i.e. Lin’s transfork has a QR code thereon having a predetermined relationship with the food item, where the code is used to determine the position and area of the food item, and the overlay area is set based thereon, where, as discussed above, Persson teaches that the virtual image associated with each marker is set at a specified offset relative to the detected marker, i.e. each marker code indicates a virtual image taking up at least a 2D area and positioned at an offset relative to the marker code.
Regarding claim 5, the limitations are similar to those treated in the above rejection(s) and are met by the references as discussed in claim 4 above, i.e. as was noted in the claim 4 rejection, the combination also teaches the limitations of claim 5, i.e. Lin’s transfork has a QR code thereon having a predetermined relationship with the food item, where the code is used to determine the position and area of the food item, and the overlay area is set based thereon, where, as discussed above, Persson teaches that the virtual image associated with each marker is set at a specified offset relative to the detected marker, i.e. each marker code indicates a virtual image taking up at least a 2D area and positioned at an offset relative to the marker code.
Regarding claim 6, the limitation “wherein the display control process displays, on the [overlay] area, an image rotated by an inclination angle corresponding to an inclination angle of the overlay area” is taught by Narumi and Persson (Narumi, e.g. sections 3.1, 3.2, figures 1, 3, teaches that the edible marker is tracked with 6DOF, i.e. 3 positional degrees of freedom and 3 rotational degrees of freedom, such that the overlaid image is rotated to match any change in angle of the overlay area. While best observed in color, Narumi’s figure 3 shows that the template image of the meta cookie with the edible marker has a reference orientation, where in the whole cookie example the overlaid image is rotated and aligned such that the dominant ridges are rotated roughly 5 degrees from vertical, and in the bitten cookie example the ridges are closer to 45 degrees from vertical, and in the divided cookie example, each cookie portion has the correspondingly rotated divided portion of the overlaid image. It is additionally noted that, as discussed in the claim 4 rejection, Persson’s markers and associated virtual image offsets also have 6DOF, such that the combined system would specify the overlay area using an offset accounting for not only the relative position between Lin’s transfork QR code and the food item speared by the fork, but relative angles, including inclination angle, between the overlay area and the HMD viewing/display orientation.)
Response to Arguments
Applicant’s arguments, see pages 6-9, filed 3/12/26, with respect to the rejection(s) of claim(s) 1-8 under 35 U.S.C. 102 and 103 in view of Narumi, Altamirano, Ueda, Lin, and Persson have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of Narumi, Ohashi, Altamirano, Ueda, Lin, and Persson.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ROBERT BADER whose telephone number is (571)270-3335. The examiner can normally be reached 11-7 m-f.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Tammy Goddard can be reached at 571-272-7773. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ROBERT BADER/Primary Examiner, Art Unit 2611
Prosecution Insights