Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-2, 4-12, 14-15, 17-18, 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Fong U.S. Patent/PG Publication 15064624.
Regarding claim 1 (independent):
One or more non-transitory computer-readable media encoded with instructions that, when executed, configure a computing device for: (Fong [0005] The host device may include a host camera and a host computer connected with the host camera. The host camera may be configured to produce the image and the host computer may be configured to process the image.)
detecting, via a sensor, a physical surface within a line of sight of the sensor (Fong [0179] 1. Setup the host camera 133 so that the camera can capture the area where the wall is going to be painted;)(Fong [0150] As shown in FIG. 12K, the calibration mark 4 includes a set of stationary tracking objects on the physical surface which defines the target surface used to reproduce the virtual object. Such a configuration may maintain the aspect ratio between the physical object and the virtual object and allow the system to automatically compensate the movement between the host camera and the target surface. The number of the stationary tracking objects is not limited. Two or more stationary tracking objects may be provided on the target surface. Also, the position of each stationary tracking object is not limited. Taking FIG. 12K as an example, two stationary tracking objects may be respectively provided on two corners of the target surface. Obviously, four stationary tracking objects may be respectively provided on four corners of the target surface to define a quadrilateral surface as shown in FIG. 12K. A tracking object carried by the detector device 3 may be movable on the target surface defined by the stationary tracking objects.).
providing a virtual surface corresponding to the physical surface (Fong [0138] The idea is to apply “3D perspective reproduction” techniques which map three-dimensional points to a two-dimensional plane as each host device represents a 2D plane. The 3D model of the known surface is first created in 3D CAD software based on the known geometry, and then the virtual cameras of the same quantity as the physical host devices are created in the virtual CAD environment. Each virtual camera is aligned in the way that the orientation and distance between the real camera and the real surface are the same as the virtual camera and the virtual surface by using known property of the calibration mark on the real surface as well as the sensor information in the host device such as distance, angular information between the host device and the target surface. After the calibration, the 3D target surface is mapped to multiple 2D planes and then we can use the same techniques as aforementioned to reproduce any object on the 3D surface.).
receiving an image from a source other than the sensor (Fong [0179] 2. Select a photo that the user wants to paint on the wall (for example a picture of The Statue of Freedom);).
dividing the image into pieces (Fong [0179] 6. Freeze the photo and generate the template (FIG. 15 illustrates the generation of the template);)(Fong [0177] The whole concept of wall art painting is to break down the whole giant artwork into small puzzles. Each puzzle has a contour line and then filled with the appropriate color. The trickiest part of wall art painting is to outline the contour of the artwork on the wall with the exact scale. Once the contour is done on the wall, everyone can complete the color filling part by themselves.)(Fong [0160] 8. Convert the selected photo into various layers such as contour layers, color layers, and etc.).
and displaying, within the virtual surface, placement indicators each denoting a location on the physical surface to place a physical object corresponding to one of the image pieces (Fong [0162] 1. Select the desired layer for drawing. [0163] 2. The selected layer will be overlaid on the live captured image.)(Fong [0003] Optical projection is sometimes used in reproducing a virtual object on a surface (2D or 3D surface) such as a wall with a projected image so that a painter can paint the wall according to the projected image.)(Fong [0182] The process of reproducing the color in the template on the wall includes the following steps: 1. The user goes to the wall with the detector device and the paints are marked with numbers; 2. The user uses the detector device 137 to locate the puzzle that he wants to paint; 3. The host computer 135 updates the color information of that particular area to the screen of the smart phone 143; 4. The user selects the appropriate color and then fills the area with the color; 5. Repeat the steps 2 to 4 until all the areas are filled with appropriate colors.)(Fong [0201] The system for reproducing virtual objects in the above embodiment can be applied to Photo Wall Layout. As illustrated in FIG. 18, in this case, it is the photo frame installed on the wall need to be leveled and in an exact scale and location as planned. The system needs to be calibrated as aforementioned. After the calibration, the objects (virtual photo frames forming a “heart” shape) are reproduced perfectly on the wall at the exact orientations and scale which form a virtual “heart” shape. Then the user can follow the reproduced virtual image to install the photo frames on the wall.) where the user can place a physical object in the form of paint or a picture frame.
Fong discloses the above elements in several embodiments. With the embodiments being disclosed in a single reference, one of ordinary skill in the art at the time of the filing of the invention being aware of one embodiment would also have been aware of the others, and it would have been obvious to one of ordinary skill in the art at the time of the filing of the invention to have combined these elements from two or more embodiments into a single arrangement for the benefit of enjoying the advantages of all the embodiments disclosed combined into a single arrangement.
Regarding claim 2:
The one or more non-transitory computer-readable media of claim 1, has all of its limitations taught by Fong. Fong further teaches comprising further instructions for detecting placement of a plurality of the physical objects (Fong [0162] 1. Select the desired layer for drawing. [0163] 2. The selected layer will be overlaid on the live captured image.)(Fong [0003] Optical projection is sometimes used in reproducing a virtual object on a surface (2D or 3D surface) such as a wall with a projected image so that a painter can paint the wall according to the projected image.)(Fong [0182] The process of reproducing the color in the template on the wall includes the following steps: 1. The user goes to the wall with the detector device and the paints are marked with numbers; 2. The user uses the detector device 137 to locate the puzzle that he wants to paint; 3. The host computer 135 updates the color information of that particular area to the screen of the smart phone 143; 4. The user selects the appropriate color and then fills the area with the color; 5. Repeat the steps 2 to 4 until all the areas are filled with appropriate colors.)(Fong [0201] The system for reproducing virtual objects in the above embodiment can be applied to Photo Wall Layout. As illustrated in FIG. 18, in this case, it is the photo frame installed on the wall need to be leveled and in an exact scale and location as planned. The system needs to be calibrated as aforementioned. After the calibration, the objects (virtual photo frames forming a “heart” shape) are reproduced perfectly on the wall at the exact orientations and scale which form a virtual “heart” shape. Then the user can follow the reproduced virtual image to install the photo frames on the wall.)
Regarding claim 4:
The one or more non-transitory computer-readable media of claim 1, has all of its limitations taught by Fong. Fong further teaches wherein at least one of the physical objects comprises more than one color (Fong [0182] The process of reproducing the color in the template on the wall includes the following steps: 1. The user goes to the wall with the detector device and the paints are marked with numbers; 2. The user uses the detector device 137 to locate the puzzle that he wants to paint; 3. The host computer 135 updates the color information of that particular area to the screen of the smart phone 143; 4. The user selects the appropriate color and then fills the area with the color; 5. Repeat the steps 2 to 4 until all the areas are filled with appropriate colors.)(Fong [0201] The system for reproducing virtual objects in the above embodiment can be applied to Photo Wall Layout. As illustrated in FIG. 18, in this case, it is the photo frame installed on the wall need to be leveled and in an exact scale and location as planned. The system needs to be calibrated as aforementioned. After the calibration, the objects (virtual photo frames forming a “heart” shape) are reproduced perfectly on the wall at the exact orientations and scale which form a virtual “heart” shape. Then the user can follow the reproduced virtual image to install the photo frames on the wall.)
Regarding claim 5:
The one or more non-transitory computer-readable media of claim 1, has all of its limitations taught by Fong. Fong further teaches wherein one of the physical objects is placed according to a placement indicator (Fong [0182] The process of reproducing the color in the template on the wall includes the following steps: 1. The user goes to the wall with the detector device and the paints are marked with numbers; 2. The user uses the detector device 137 to locate the puzzle that he wants to paint; 3. The host computer 135 updates the color information of that particular area to the screen of the smart phone 143; 4. The user selects the appropriate color and then fills the area with the color; 5. Repeat the steps 2 to 4 until all the areas are filled with appropriate colors.)(Fong [0201] The system for reproducing virtual objects in the above embodiment can be applied to Photo Wall Layout. As illustrated in FIG. 18, in this case, it is the photo frame installed on the wall need to be leveled and in an exact scale and location as planned. The system needs to be calibrated as aforementioned. After the calibration, the objects (virtual photo frames forming a “heart” shape) are reproduced perfectly on the wall at the exact orientations and scale which form a virtual “heart” shape. Then the user can follow the reproduced virtual image to install the photo frames on the wall.)
Regarding claim 6:
The one or more non-transitory computer-readable media of claim 5, has all of its limitations taught by Fong. Fong further teaches wherein the placement indicator corresponds to a color, pattern of colors, or shape of the one of the physical objects (Fong [0182] The process of reproducing the color in the template on the wall includes the following steps: 1. The user goes to the wall with the detector device and the paints are marked with numbers; 2. The user uses the detector device 137 to locate the puzzle that he wants to paint; 3. The host computer 135 updates the color information of that particular area to the screen of the smart phone 143; 4. The user selects the appropriate color and then fills the area with the color; 5. Repeat the steps 2 to 4 until all the areas are filled with appropriate colors.)(Fong Fig. 16)
Regarding claim 7:
The one or more non-transitory computer-readable media of claim 6, has all of its limitations taught by Fong. Fong further teaches wherein each physical object has a single color and the placement indicators are numbers that correspond to the colors of the physical objects (Fong [0182] The process of reproducing the color in the template on the wall includes the following steps: 1. The user goes to the wall with the detector device and the paints are marked with numbers; 2. The user uses the detector device 137 to locate the puzzle that he wants to paint; 3. The host computer 135 updates the color information of that particular area to the screen of the smart phone 143; 4. The user selects the appropriate color and then fills the area with the color;).
Regarding claim 8:
The one or more non-transitory computer-readable media of claim 1, has all of its limitations taught by Fong. Fong further teaches wherein virtual pieces are separated by boundaries (Fong Fig. 16)(Fong [0177] The whole concept of wall art painting is to break down the whole giant artwork into small puzzles. Each puzzle has a contour line and then filled with the appropriate color. The trickiest part of wall art painting is to outline the contour of the artwork on the wall with the exact scale. Once the contour is done on the wall, everyone can complete the color filling part by themselves.).
Regarding claim 9:
The one or more non-transitory computer-readable media of claim 8, has all of its limitations taught by Fong. Fong further teaches wherein two of the boundaries forma non-orthogonal angle (Fong Fig. 16)(Fong [0177] The whole concept of wall art painting is to break down the whole giant artwork into small puzzles. Each puzzle has a contour line and then filled with the appropriate color. The trickiest part of wall art painting is to outline the contour of the artwork on the wall with the exact scale. Once the contour is done on the wall, everyone can complete the color filling part by themselves.).
Regarding claim 10:
The one or more non-transitory computer-readable media of claim 1, has all of its limitations taught by Fong. Fong further teaches wherein the physical surface has a curvature or comprises an intersection of two planes (Fong [0137] FIG. 12I illustrates a system for reproducing virtual objects according to another embodiment of the present patent application. Referring to FIG. 12I, the system is constructed with multiple host devices that can be used to reproduce an object on a 3D surface whose geometry is previously known. Referring to FIG. 12I, 1a, 1b, 1c, 1d, le, and if are the host devices. The number of the host devices required depends on the real application and can be any number greater than 1. 2 is a 3D surface (or sphere). 3 are the calibration marks. 4 is the user with the detector device.)(Fong 17h, 23g)(Fong [0193] With the aid of the computer navigated machine, computer navigated drawing can be achieved on any surface including vertical wall, ceiling, exterior wall of any building and etc.).
Regarding claim 11:
The one or more non-transitory computer-readable media of claim 1, has all of its limitations taught by Fong. Fong further teaches comprising further instructions for:
displaying the image upon the virtual surface based upon the sensor being a distance from the physical surface and transforming, on the virtual surface, the image into the placement indicators based upon a change in the distance from sensor to the physical surface (Fong [0078] The calibration sensor 209 is configured to provide addition information to the computer unit 205 to enhance the application. The calibration sensor 209 may be a GPS unit, a level sensor, a gyroscope, a proximity sensor, a distance sensor and etc.)(Fong [0151] The setup process of the system includes the following steps: [0152] 1. Connect the system depicted in FIG. 12K. [0153] 2. Launch the software in the PC. [0154] 3. Align the camera until the FOV covers all the calibration mark on the surface. [0155] 4. Correct the angular error (pitch, yaw and roll) using the calibration mark by software algorithm [0156] In this case, the main error is the pitch error since the camera is not in parallel to the surface. The rectangular surface will become a trapezoid surface, as illustrated in FIG. 12M. Referring to FIG. 12M, A is the rectangular surface, B is the trapezoid image captured by the camera, and C is the corrected rectangular image using the calibration mark. [0157] 5. Load the selected photo. [0158] 6. Overlay the selected photo on the captured image. [0159] 7. Scale and reposition the overlaid image to the user's desired form. [0160] 8. Convert the selected photo into various layers such as contour layers, color layers, and etc.)(Fong [0138] The idea is to apply “3D perspective reproduction” techniques which map three-dimensional points to a two-dimensional plane as each host device represents a 2D plane. The 3D model of the known surface is first created in 3D CAD software based on the known geometry, and then the virtual cameras of the same quantity as the physical host devices are created in the virtual CAD environment. Each virtual camera is aligned in the way that the orientation and distance between the real camera and the real surface are the same as the virtual camera and the virtual surface by using known property of the calibration mark on the real surface as well as the sensor information in the host device such as distance, angular information between the host device and the target surface. After the calibration, the 3D target surface is mapped to multiple 2D planes and then we can use the same techniques as aforementioned to reproduce any object on the 3D surface.).
Regarding claim 12:
The one or more non-transitory computer-readable media of claim 1, has all of its limitations taught by Fong. Fong further teaches comprising further instructions for snapping a virtual representation of a physical object onto the virtual surface (Fong [0080] FIG. 4 illustrates the operation of the system for reproducing virtual objects in this embodiment in reconstructing a blue virtual object (0D object) on the same surface as the physical red dot. Referring to FIG. 3 and FIG. 4, the red dot 301 (shown as 401 in the part A of FIG. 4) on the tracking object 300 is sensed by digital light sensor 203 and transformed by the computer unit 205 to a virtual tracking object represented by a matrix (shown as 403 in the part B of FIG. 4). The matrix can be 0D, 1D, 2D, or 3D depending on the type of sensor being used. The matrix is further transformed to a 0D object (for this illustration, the resolution of the 0D object is ONE unit of the matrix, and it can be lower than ONE unit by using a sub-pixel estimation algorithm) so that the red dot 301 is logically represented by a coordinate in either the physical space or virtual space. When the detector device 103 moves, the tracking object 300 moves, the red dot 301 moves, and the coordinates of the red dot 301 in the virtual space will change as well.)(Fong [0008] The host device may be configured to scale, rotate or relocate the template pattern in the virtual space in manipulating the template pattern. The host device may be configured to manipulate the template pattern based on the user's perception. )(Fong [0156] In this case, the main error is the pitch error since the camera is not in parallel to the surface. The rectangular surface will become a trapezoid surface, as illustrated in FIG. 12M. Referring to FIG. 12M, A is the rectangular surface, B is the trapezoid image captured by the camera, and C is the corrected rectangular image using the calibration mark. [0157] 5. Load the selected photo. [0158] 6. Overlay the selected photo on the captured image. [0159] 7. Scale and reposition the overlaid image to the user's desired form. [0160] 8. Convert the selected photo into various layers such as contour layers, color layers, and etc.).
Regarding claim 14 (independent):
The claim is a parallel version of claim 1. As such it is rejected under the same teachings.
Regarding claim 15:
The claim is a parallel version of claim 2. As such it is rejected under the same teachings.
Regarding claim 17 (independent):
The claim is a parallel version of claim 1. As such it is rejected under the same teachings.
Regarding claim 18:
The claim is a parallel version of claim 2. As such it is rejected under the same teachings.
Regarding claim 20:
The claim is a parallel version of claim 4. As such it is rejected under the same teachings.
Claim(s) 13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Fong U.S. Patent/PG Publication 15064624 in view of Maciocci U.S. Patent/PG Publication 20120249741.
Regarding claim 13:
The one or more non-transitory computer-readable media of claim 1, has all of its limitations taught by Fong. Fong further teaches wherein providing a virtual surface further comprises an augmented reality display such that the virtual surface remains overlain upon the physical surface as the sensor updates a location of the physical surface (Fong [0129] In applications of augmented reality or virtual reality,)(Fong [0134] The AR mark and fiduciary mark become a group of pixels in the LCD display, or any other active devices that can display the tracking pattern, or a mix of passive and active pattern to form the tracking pattern.)(Fong [0158] 6. Overlay the selected photo on the captured image.).
Fong discloses an AR device describe above. However, for the purposes of compact prosecution and for further clarity, in a related field of endeavor, Maciocci teaches:
wherein providing a virtual surface further comprises an augmented reality display such that the virtual surface remains overlain upon the physical surface as the sensor updates a location of the physical surface (Maciocci Abstract A head mounted device provides an immersive virtual or augmented reality experience for viewing data and enabling collaboration among multiple users. Rendering images in a virtual or augmented reality system may include capturing an image and spatial data with a body mounted camera and sensor array, receiving an input indicating a first anchor surface, calculating parameters with respect to the body mounted camera and displaying a virtual object such that the virtual object appears anchored to the selected first anchor surface. Further operations may include receiving a second input indicating a second anchor surface within the captured image that is different from the first anchor surface, calculating parameters with respect to the second anchor surface and displaying the virtual object such that the virtual object appears anchored to the selected second anchor surface and moved from the first anchor surface.)
Therefore, it would have been obvious before the effective filing date of the claimed invention to use an AR device as taught by Maciocci. The rationale for doing so would have been that it is a simple substitution of one known element for another to obtain predictable results, where Fong projects an image on a wall for a user, and Maciocci projects an image on the wall for a user, where it is merely the type of projector that is different, where the end result for both is still displaying an image to a user . Therefore it would have been obvious to combine Maciocci with Fong to obtain the invention.
Claim(s) 3, 16, 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Fong U.S. Patent/PG Publication 15064624 in view of Gray U.S. Patent/PG Publication 9965895.
Regarding claim 3:
The one or more non-transitory computer-readable media of claim 2, has all of its limitations taught by Fong. Fong does not teach error checking. In a related field of endeavor, Gray teaches:
comprising further instructions for:
comparing placement of the plurality of the physical objects to confirm correct placements according to the placement indicators and outputting an indication of a physical object having a correct or an incorrect placement upon the placement surface (Gray C 8 L60-C9 L20 In accordance with various embodiments, the level of progress can be shown in a number of different ways, including a progress bar, text, or some other textual or graphical display. For example, the color of the outline of the overlaid representation of the scene can indicate areas drawn and areas yet drawn. In this example, red lines can correspond to areas of the image that have not been sketched and green lines correspond to areas that have been sketched. In another example, comparing the model to the source image can include, analyzing a similarity between corresponding metrics between the model and the source image. The metrics can include, for example, perspective, ratio, alignment, etc. as relating to the model and the source image. In yet another example, a number of errors or other inconsistencies can be displayed to the user and an indication 420 emphasizing or otherwise pointing out the errors can be displayed on the computing device. For example, in at least one embodiment, the model can be compared to the source image, and at least one prompt can be generated based on the comparison, the at least one prompt including information indicating at least one or a progress in creating the scene or the object, a number of errors in creating the scene or the object, an indication of an error in creating the scene or the object.)
Therefore, it would have been obvious before the effective filing date of the claimed invention to use error checking as taught by Gray. The motivation for doing so would have been to find errors and inconsistencies, improving the quality and preventing mistakes (Gray C 8 L60-C9 L20). Therefore it would have been obvious to combine Gray with Fong to obtain the invention.
Regarding claim 16:
The claim is a parallel version of claim 3. As such it is rejected under the same teachings.
Regarding claim 19:
The claim is a parallel version of claim 3. As such it is rejected under the same teachings.
Conclusion
For the prior art referenced and the prior art considered pertinent to Applicant’s disclosure but not relied upon, see PTO-892 “Notice of References Cited”.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JASON PRINGLE-PARKER whose telephone number is (571) 272-5690 and e-mail is jason.pringle-parker@uspto.gov. The examiner can normally be reached on 8:30am-5:00pm est Monday-Friday. If attempts to reach the examiner by telephone are unsuccessful, the examiner's supervisor, King Poon can be reached on (571) 270-0728. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, seehttp://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JASON A PRINGLE-PARKER/
Primary Examiner, Art Unit 2617
