Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-4, and 10-11 are rejected under 35 U.S.C. 103 as being unpatentable over Mullins et al (US 9659381 B2, hereinafter Mullins) and Hare (US 11189098 B2, hereinafter Hare).
Regarding claim 1, Mullins teaches An information processing method in a computer, the method comprising: acquiring a three-dimensional object by reproducing a real object including a dynamic region in a virtual space (Col 3 Line 61-64 “The texture mapping module detects changes in the texture in the image of the real-world object, identifies portions of the image of the real-world object with texture changes”, Col 11 Line 11-13 “the AR visualization module 406 may retrieve three-dimensional models of virtual objects associated with a captured real-world object.”); acquiring a real image of the real object captured by an imaging device (Col 2 Line 62-63 “The optical sensor captures an image of a real-world object or image.”); detecting a second dynamic region indicating the dynamic region in the real image (Col 3 Line 61-64 “The texture mapping module detects changes in the texture in the image of the real-world object, identifies portions of the image of the real-world object with texture changes”, Col 8 Line 53-55 “The predefined area recognition module 302 identifies one or more predefined area in the image of the real-world object.”); embedding an image of the second dynamic region in real time as a texture image of the first dynamic region (Col 3 Line 7-11 “The texture mapping module then maps the texture extracted from the image of the real-world object to a virtual object associated with the real-world object. For example, the extracted texture may be mapped to a texture of a three-dimensional model of the virtual object.”); and outputting a display image of the three-dimensional object in which the image of the second dynamic region has been embedded (Col 3 Line 42-47 “The texture mapping module retrieves a virtual object corresponding to the identified real-world object, maps the texture to the virtual object, dynamically updates the texture to the virtual object in real time, and generates a visualization of the virtual object in a display of the viewing device.”).
Mullins teaches mapping a texture from an area of an image to a corresponding area of a three-dimensional object (Col 9 Line 22-25 “the texture mapping module 220 maps the texture of a predefined area in the image of the real-world object to a corresponding area of the virtual object.”), but fails to explicitly teach identifying a first dynamic region indicating the dynamic region in the three-dimensional object. In related field of endeavor, Hare teaches identifying a first dynamic region indicating the dynamic region in the three-dimensional object (Col 19 Line 6-9 “the rendering component 502 may identify the region of the virtual object dedicated to displaying the images from the second camera feed”).
It would have been obvious to one of ordinary skill in the art prior to the time of filing to have modified Mullins to include identifying a first dynamic region indicating the dynamic region in the three-dimensional object as taught by Hare. Doing so would improve functionality of electronic messaging and imaging software by allowing users to modify textures of an object (Col 2 Line 28-32 “embodiments of the present disclosure improve the functionality of electronic messaging and imaging software and systems by providing functionality that allows users to modify surface textures of a virtual object”).
Regarding claim 2, Mullins as modified by Hare teaches the information processing method according to claim 1, and Mullins further teaches wherein the real object has a marker indicating the dynamic region (Col 8 Line 59-61 “the predefined area may include specific markings (e.g., QR codes or unique patterns) in specific areas (e.g., corners of a page)”), and the detecting the second dynamic region includes detecting the second dynamic region based on an image indicating the marker included in the real image (Col 8 Line 53-59 “The predefined area recognition module 302 identifies one or more predefined area in the image of the real-world object ... The predefined area recognition module 302 identifies the outline of the cartoon character as a predefined area. In another example, the predefined area may include specific markings (e.g., QR codes or unique patterns) in specific areas (e.g., corners of a page).”).
Mullins teaches detecting a region based on the marker (Col 8 Line 53-59 “The predefined area recognition module 302 identifies one or more predefined area in the image of the real-world object ... The predefined area recognition module 302 identifies the outline of the cartoon character as a predefined area. In another example, the predefined area may include specific markings (e.g., QR codes or unique patterns) in specific areas (e.g., corners of a page).”), but fails to teach identifying the first dynamic region in the three-dimensional object. In related field of endeavor, Hare teaches identifying the first dynamic region in the three-dimensional object (Col 19 Line 6-9 “the rendering component 502 may identify the region of the virtual object dedicated to displaying the images from the second camera feed”).
It would have been obvious to one of ordinary skill in the art prior to the time of filing to have further modified Mullins and Hare to include identifying the first dynamic region in the three-dimensional object as taught by Hare. Doing so would improve functionality of electronic messaging and imaging software by allowing users to modify textures of an object (Col 2 Line 28-32 “embodiments of the present disclosure improve the functionality of electronic messaging and imaging software and systems by providing functionality that allows users to modify surface textures of a virtual object”).
Regarding claim 3, Mullins as modified by Hare teaches the information processing method according to claim 1, and Mullins further teaches wherein the dynamic region corresponds to an object element constituting the real object (Col 3 Line 61-64 “The texture mapping module detects changes in the texture in the image of the real-world object, identifies portions of the image of the real-world object with texture changes”, Col 8 Line 53-55 “The predefined area recognition module 302 identifies one or more predefined area in the image of the real-world object.”), and the detecting the second dynamic region includes detecting a region of the object element in the real image with the object recognition processing (Col 8 Line 7-14 “ The recognition module 214 identifies the object at which the viewing device 101 is pointed. The recognition module 214 may detect, generate, and identify identifiers such as feature points of the physical object being viewed or pointed at by the viewing device 101 using an optical device of the viewing device 101 to capture the image of the physical object”, Col 8 Line 53-55 “The predefined area recognition module 302 identifies one or more predefined area in the image of the real-world object”, Col 13 Line 27-30 “the viewing device 101 identifies the object A 116 using a machine vision recognition algorithm or based on predefined unique markings (e.g., QR codes, unique patterns) in predefined regions”). Mullins teaches identifying a region with object recognition processing (Col 8 Line 53-55 “The predefined area recognition module 302 identifies one or more predefined area in the image of the real-world object”, Col 13 Line 27-30 “the viewing device 101 identifies the object A 116 using a machine vision recognition algorithm or based on predefined unique markings (e.g., QR codes, unique patterns) in predefined regions”), but fails to explicitly teach identifying the first region of the object element in the three-dimensional object. In related field of endeavor, Hare further teaches identifying the first dynamic region of the object element in the three-dimensional object (Col 19 Line 6-9 “the rendering component 502 may identify the region of the virtual object dedicated to displaying the images from the second camera feed”).
It would have been obvious to one of ordinary skill in the art prior to the time of filing to have further modified Mullins and Hare to include identifying the first region of the object element in the three-dimensional object as taught by Hare. Doing so would improve functionality of electronic messaging and imaging software by allowing users to modify textures of an object (Col 2 Line 28-32 “embodiments of the present disclosure improve the functionality of electronic messaging and imaging software and systems by providing functionality that allows users to modify surface textures of a virtual object”).
Regarding claim 4, Mullins as modified by Hare teaches the information processing method according to claim 1, and Mullins further teaches wherein the real image is repeatedly acquired, and the detecting, the embedding, and the outputting of the second dynamic region are performed each time the real image is acquired (Col 9 Line 46-58 “The AR content real-time texture mapping module 404 detects changes in the texture in the image of the real-world object. For example, the AR content real-time texture mapping module 404 identifies portions of the image of the real-world object with texture changes and dynamically updates in real time a mapping of texture to parts of the virtual content corresponding to the portions of the image with texture changes. In another example, the texture extraction module 218 extracts a texture of an image of the real-world object on a periodic basis. The texture mapping module 220 updates a mapping of the texture to the virtual object in response to detecting changes in the texture of the image of the real-world object.”)
Regarding claim 10, the information processing device claim 10 is similar in scope to the method claim 1, and is rejected under similar rationale (Mullins Col 12 Line 28-29 “Any one or more of the modules described herein may be implemented using hardware (e.g., a processor of a machine)”).
Regarding claim 11, the non-transitory computer readable recording medium claim 11 is similar in scope to the method claim 1, and is rejected under similar rationale (Mullins Col 17 Line 64-67 “embodiments may be implemented using a computer program product, e.g., a computer program tangibly embodied in an information carrier, e.g., in a machine-readable medium”).
Claims 5-6 are rejected under 35 U.S.C. 103 as being unpatentable over Mullins and Hare as applied to claim 1 above, and further in view of Efrat et al (US 7423673 B1, hereinafter Efrat).
Regarding claim 5, Mullins as modified by Hare teaches The information processing method according to claim 1, and Mullins further teaches embedding the image of the second dynamic region after the correcting in the first dynamic region (Col 3 Line 7-11 “The texture mapping module then maps the texture extracted from the image of the real-world object to a virtual object associated with the real-world object. For example, the extracted texture may be mapped to a texture of a three-dimensional model of the virtual object.”) but fails to explicitly teach wherein the embedding the image of the second dynamic region includes acquiring a distortion parameter of the imaging device, and correcting distortion of the image of the second dynamic region using the distortion parameter.
In related field of endeavor, Efrat teaches acquiring a distortion parameter of the imaging device (Col 11 Line 34-39 “image processor 190 determines a distortion parameter, such as displacement D.sub.2 between locations 216 and 220, deviation angle .alpha..sub.2, and the like, respective of the apparent location of feature 208 and location 216. Image processor 190 determines distortion parameters with respect to additional features of object 206.”), and correcting distortion of the image of the second dynamic region using the distortion parameter (Col 8 Line 4-6 “the image processor can make corrections to the image, due to distortions caused by these optical elements.”). It would have been obvious to one of ordinary skill in the art prior to the time of filing to have further modified Mullins and Hare to include acquiring a distortion parameter of the imaging device and correcting distortion of the image of the second dynamic region using the distortion parameter as taught by Efrat. Doing so would allow corrections to the image to be made due to distortions (Col 8 Line 4-6 “the image processor can make corrections to the image, due to distortions caused by these optical elements”)
Regarding claim 6, Mullins, Hare and Efrat teach the information processing method according to claim 5, and Efrat further teaches wherein the distortion parameter is estimated by camera calibration using current position and attitude information about the imaging device, initial position and attitude information about the imaging device at a time of capturing an initial texture image embedded in the first dynamic region (Col 6 Line 28-31 “The term "position" of an object herein below, refers to either the location or the orientation of the object, or both the location and orientation thereof”, Col 10 Line 36-38 “a relative position between the first image acquisition position and a second image acquisition position is determined”, Col 10 Line 43-48 “a first dynamic distortion image is acquired from the first image acquisition position, and a second dynamic distortion image is acquired from the second image acquisition position, both the first dynamic distortion image and the second dynamic distortion image being acquired with respect to the same object”), two-dimensional position information about the dynamic region in the initial texture image, two-dimensional position information about the second dynamic region in the real image (Col 11 Line 20-23 “a dynamic distortion parameter is determined according to each of the apparent locations, the corresponding second dynamic distortion image feature, and the relative position”, Col 11 Line 33-38 “image processor 190 determines a distortion parameter, such as displacement D.sub.2 between locations 216 and 220, deviation angle .alpha..sub.2, and the like, respective of the apparent location of feature 208 and location 216. Image processor 190 determines distortion parameters with respect to additional features of object 206”), and three-dimensional position information about the first dynamic region (Col 12 Line 18-23 “The coordinates of different features of target 278, such as that of feature 280, are known in a coordinate system I. The image processor stores the values of the coordinates of feature 280 along the X and Y axes of coordinate system I, and distance S.sub.1 along the Z axis of coordinate system I”, Col 13 Line 14-18 “The image processor determines a distortion model for window 274, by repeating the above procedures for other features of targets 278 and 284. First, image sensor 272 acquires a first set of images of a first set of features of target 278 at location L.sub.1”).
It would have been obvious to one of ordinary skill in the art prior to the time of filing to have further modified Mullins, Hare, and Efrat to include wherein the distortion parameter is estimated by camera calibration using current position and attitude information about the imaging device, initial position and attitude information about the imaging device at a time of capturing an initial texture image embedded in the first dynamic region, two-dimensional position information about the dynamic region in the initial texture image, two-dimensional position information about the second dynamic region in the real image, and three-dimensional position information about the first dynamic region as taught by Efrat. Doing so would allow corrections to the image to be made due to distortions (Col 8 Line 4-6 “the image processor can make corrections to the image, due to distortions caused by these optical elements”)
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Mullins, Hare, and Efrat as applied to claim 6 above, and further in view of Carter et al (US 11019283 B2, hereinafter Carter).
Regarding claim 7, Mullins, Hare and Efrat teach The information processing method according to claim 6, but fail to explicitly teach wherein the current position and attitude information about the imaging device is estimated by applying a self-position estimation algorithm to the initial texture image and the real image. In related field of endeavor, Carter teaches wherein the current position and attitude information about the imaging device is estimated by applying a self-position estimation algorithm to the initial texture image and the real image. (Col 11 Line 25-32 “ the computing system may optionally estimate a three-dimensional (3D) position and pose of the real world camera at the time that the camera captured the given frame. The computing system may employ one or more of various camera solving techniques known in the field of computer vision. The result may be a “six degrees of freedom” (6DoF) estimation that includes 3D estimated (x, y, z) coordinates and a 3D rotation of the camera.”) It would have been obvious to one of ordinary skill in the art prior to the time of filing to have further modified Mullins, Hare, and Efrat to include wherein the current position and attitude information about the imaging device is estimated by applying a self-position estimation algorithm to the initial texture image and the real image as taught by Carter. Doing so would allow augmented content to have a position and pose of the surface it is fit to (Col 12 Line 5-10 “the computing system may use this data when augmenting the frame with an advertisement or other augmentation content in order for the augmentation content to have a position and pose that matches the position and pose of the surface fit to one or more candidate regions above”)
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Mullins and Hare as applied to claim 1 above, and further in view of Fox (US 10891713 B2). Regarding claim 8, Mullins and Hare teach the information processing method according to claim 1, and Mullins further teaches embedding the edited image of the second dynamic region in the first dynamic region (Col 3 Line 7-11 “The texture mapping module then maps the texture extracted from the image of the real-world object to a virtual object associated with the real-world object. For example, the extracted texture may be mapped to a texture of a three-dimensional model of the virtual object.”), and mapping the texture of an area of an image to a corresponding area of a virtual object (Col 3 Line 53-55 “The texture mapping module may map the texture of a predefined area in the image of the real-world object to a corresponding area of the virtual object”), but fails to explicitly teach wherein the embedding the image of the second dynamic region includes editing the image of the second dynamic region to match with a shape of the first dynamic region, based on two-dimensional position information about a vertex of the second dynamic region and two-dimensional position information about a vertex of the first dynamic region.
In related field of endeavor, Fox teaches editing the image of the second dynamic region to match with a shape of the first dynamic region, based on two-dimensional position information about a vertex of the second dynamic region and two-dimensional position information about a vertex of the first dynamic region (Col 2 Line 25-30 “take a two-dimensional (2D) representation of an image and transform it, adjusting the aspect ratio, vertices and alignment to map the image onto a three dimensional (3D) format, such as for a conical object, for example a paper cup. The process provides the ability to transpose rectangular grid coordinates onto a curved conical shape”)
It would have been obvious to one of ordinary skill in the art prior to the time of filing to have further modified Mullins and Hare to include editing the image of the second dynamic region to match with a shape of the first dynamic region, based on two-dimensional position information about a vertex of the second dynamic region and two-dimensional position information about a vertex of the first dynamic region as taught by Fox. Doing so would allow the image to appear perfectly rendered and aligned (Col 3 Line 16-19 “When the 2D transformed representation of the image is applied to the conical shape, the image will appear to be perfectly rendered and aligned”)
Claims 9 is rejected under 35 U.S.C. 103 as being unpatentable over Mullins and Hare as applied to claim 1 above, and further in view of Jacquement et al (US 10205889 B2, hereinafter Jacquement).
Regarding claim 9, Mullins and Hare teach the information processing method according to claim 1, but fail to explicitly teach wherein the dynamic region includes a region corresponding to a display region of a monitor included in the real object. In related field of endeavor, Jacquement teaches wherein the dynamic region includes a region corresponding to a display region of a monitor included in the real object (Col 1 Line 20-23 “A typical application of the present method may be used during the live broadcast of sporting events to replace advertisement images appearing on perimeter boards surrounding the play field or other areas at the venue”, Col 3 Line 63-66 “The playground is surrounded with perimeter boards or electronic billboards 2 on which static or dynamic advertisement images may be displayed”)
It would have been obvious to one of ordinary skill in the art prior to the time of filing to have further modified Mullins and Hare to include wherein the dynamic region includes a region corresponding to a display region of a monitor included in the real object as taught by Jacquement. Doing so would allow variation in the appearance of these areas (Col 1 Line 23-26 “This allows the delivery of multiple advertisement content appearing on these areas when broadcasting to different locations.”)
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Waruna et al (US 20220092858 A1, hereinafter Waruna) teaches
Van Hoff et al (US 11217006 B2, hereinafter van Hoff) teaches creating a 3D model of an object from 2D images, and compressing and texturizing the 3D model.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JOHN PATRICK GOCO whose telephone number is (571)272-5872. The examiner can normally be reached M-Th, 7:00 am - 5:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee Tung can be reached at (571)272-7794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/J.P.G./Examiner, Art Unit 2611
/KEE M TUNG/Supervisory Patent Examiner, Art Unit 2611
Prosecution Insights