DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1 is/are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Zhang (see citation below).
Zhang teaches a method of training a machine vision system comprising a convolutional neural network (2.2 Convolutional Neural Network - CNN has been successfully applied to many computer vision tasks, p. 2; train the networks, p. 7, 5th paragraph), the method comprising: generating or selecting a set of fiducial tags (4.3 Training Data Preparation – Binarized markers are randomly sampled from several predefined marker families, p. 7; DeepTag is a general CNN-based fiducial marker detector, p. 3, left column, last sentence); generating a set of transformed fiducial tags by randomly transforming each fiducial tag from the set of fiducial tags (4.3 Training Data Preparation – Then, random grayscale, spotlighting effect, Gaussian blur, Gaussian noise and motion blur are added to simulate various printing conditions, lighting conditions and marker movements, p. 7); generating a set of superimposed training images by superimposing each transformed fiducial tag from the set of transformed fiducial tags on random image from a set of images (Fig. 10, 4.3 Training Data Preparation – Augmented markers are added into random locations with quadrilateral boundaries (Fig. 10(a-b)), p. 7); and training the convolutional neural network by processing each superimposed training image in the set of superimposed training images (4.3 Training Data Preparation describes that synthetic image is used to train the networks, p. 7; the network is the CNN, 2.2 Convolutional Neural Network, p. 2).
Claim(s) 11 is/are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Kruse (US 20230384114A1).
Kruse teaches a system for identifying a fiducial associated with an entity, the system comprising: a set of fiducial tags (21), wherein each fiducial tag from the set of fiducial tags is associated with an entity from a set of entities (identify a fiducial marker, [0034], fiducial markers 21, fiducial marker 21 indicating one or more landmark features in environment, the one or more landmark features may include an object, [0043]); an image sensor (sensors for sensing including an image capture device, [0040]) configured to digital capture images of the entity and the fiducial associated with the entity (identify fiducial markers in environment using sensor data (e.g., thermal image data), [0103]); and an application server in electronic communication with the image sensor (application servers provide runtime environments for execution of services, [0070], [0103]), the application server comprising: a processor and a memory (functions may be stored on, as instructions, a computer-readable medium and executed by a hardware-based processing unit, [0268]); a machine learning module (machine learning, [0171]); an image processing module (process sensor data that includes the thermal image data, [0057]); and a fiducial identification module [0034].
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 2 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zhang (see citation below) in view of Csaszar (US 20130076522A1).
Zhang is relied upon for the teachings as discussed above relative to Claim 1.
However, Zhang does not expressly teach wherein the set of fiducial tags comprise four-character tags, six-character tags, eight-character tags, or ten-character tags. However, Csaszar teaches wherein the set of fiducial tags comprise four-character tags, six-character tags, eight-character tags, or ten-character tags (each tag comprises fiducials called markers, [0061], UTF-8 characters, [0198]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Zhang so that the set of fiducial tags comprise four-character tags, six-character tags, eight-character tags, or ten-character tags as suggested by Casaszar. It is well-known in the art to use characters for fiducial markers that need to be read and understood by people.
Claim(s) 3-4 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zhang (see citation below) in view of Rice (see citation below).
As per Claim 3, Zhang is relied upon for the teachings as discussed above relative to Claim 1.
However, Zhang does not teach wherein a subset of characters on each of the set of fiducial tags comprises error-control coding. However, Rice teaches wherein a subset of characters on each of the set of fiducial tags comprises error-control coding (fiducial tags, tags carrying symbolic data which allows existing coding techniques to achieve robust codes, an error-correcting coding scheme is presented for carrying arbitrary symbolic data in a dependable vision system, p. 259, Abstract).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Zhang so that a subset of characters on each of the set of fiducial tags comprises error-control coding because Rice suggests the advantage of giving the ability to detect or recover from image noise (p. 261, 2nd paragraph).
As per Claim 4, Zhang does not teach wherein the error-control coding comprises Reed-Solomon encoding. However, Rice teaches wherein the error-control coding comprises Reed-Solomon encoding (Reed-Solomon code, p. 266, 3rd paragraph; p. 259, Abstract).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Zhang so that the error-control coding comprises Reed-Solomon encoding as suggested by Rice. It is well-known in the art that Reed-Solomon encoding has the advantage of providing strong data integrity by correcting multiple symbol errors in a block.
Claim(s) 5 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zhang (see citation below) in view of Gu (see citation below).
Zhang is relied upon for the teachings as discussed above relative to Claim 1.
However, Zhang does not teach wherein the convolutional neural network comprises a DenseNet convolutional neural network. However, Gu teaches wherein the convolutional neural network comprises a DenseNet convolutional neural network (CNN architecture, DenseNet, p. 285, 2nd to last paragraph; implantation of fiducial markers, enabling retraining of the CNN, p. 289, 2nd paragraph).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Zhang so that the convolutional neural network comprises a DenseNet convolutional neural network as suggested by Gu. It is well-known in the art that DenseNet has the advantage of mitigating the vanishing gradient problem, making them easier to train.
Claim(s) 6 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zhang (see citation below) and Gu (see citation below) in view of Li (US 20210365716A1).
Zhang and Gu are relied upon for the teachings as discussed above relative to Claim 5.
However, Zhang and Gu do not teach wherein the DenseNet convolutional neural network comprises five Dense Blocks and one linearization block. However, Li teaches wherein the DenseNet convolutional neural network comprises Dense Blocks (dense blocks were introduced in DenseNet, [0027], CNN, [0022]) and linearization block (pre-processing necessary to convert raw image data into data that is compatible with input for the neural network, the pre-processing may include linearization, [0078]). It would have been obvious to one of ordinary skill in the art that the user can use the number of blocks as desired in order to achieve the results that are desired by the user. Thus, it would be obvious that Li comprises five Dense Blocks and one linearization block. This would be obvious for the reasons given in the rejection for Claim 5.
Claim(s) 7 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zhang (see citation below), Gu (see citation below), and Li (US 20210365716A1) in view of Buttler (see citation below).
Zhang, Gu, and Li are relied upon for the teachings as discussed above relative to Claim 6.
However, Zhang, Gu, and Li do not teach wherein the linearization block comprises four convolutions used on a single feature volume to extract four probability vectors, one for each of the four characters of the fiducial tags. However, Buttler teaches wherein the linearization block (linear model, p. 11, right column, 3rd paragraph) comprises convolutions (nine convolutions, p. 13, left column, last paragraph) used on a single feature volume to extract probability vectors (probability vectors, p. 12, 4th paragraph), one for each of the characters of the fiducial tags (fiducial markers, p. 10, 2nd paragraph). It would have been obvious to one of ordinary skill in the art that the user can use the number of characters of the fiducial tags as desired in order to achieve the results that are desired by the user. Thus, it would be obvious that Buttler comprises four convolutions to extract four probability vectors, one for each of the four characters of the fiducial tags.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Zhang, Gu, and Li so that the linearization block comprises four convolutions used on a single feature volume to extract four probability vectors, one for each of the four characters of the fiducial tags as suggested by Buttler. It is well-known in the art that using convolution with probability vectors is a fundamental building block for many machine learning tasks, such as image processing and feature extraction.
Claim(s) 8-10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zhang (see citation below) in view of Zin (see citation below).
As per Claim 8, Zhang is relied upon for the teachings as discussed above relative to Claim 1.
However, Zhang does not expressly teach wherein training the convolutional neural network further comprises training the convolutional neural network to identify a set of characters on a physical tag. However, Zin teaches wherein training the convolutional neural network further comprises training the convolutional neural network to identify a set of characters on a physical tag (3.4.4. Ear Tag Recognition, convolution neural network (CNN) is applied in the recognition step, the CNN is trained, 10,000 digits are used as training data, in the training process, the individual digits specifically used for training are manually cropped from the video data, p. 11).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Zhang so that training the convolutional neural network further comprises training the convolutional neural network to identify a set of characters on a physical tag because Zin suggests that after the CNN is trained to identify characters on a physical tag, then it will be able to more accurately identify the characters on the physical tag (3.4.4. Ear Tag Recognition, p. 11).
As per Claim 9, Zhang does not teach further comprising associating the set of characters on the physical tag with an entity. However, Zin teaches further comprising associating the set of characters on the physical tag with an entity (every calf gets a unique ID number, p. 1, 1. Introduction; Figure 1 shows the physical tag that has the unique ID number on it, p. 2).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Zhang to include associating the set of characters on the physical tag with an entity because Zin suggests that this makes it more efficient to monitor cows (1. Introduction, p. 1-2).
As per Claim 10, Zhang does not teach wherein the entity is a porcine or bovine animal. However, Zin teaches wherein the entity is a porcine or bovine animal (tracking system for individual cows using an ear tag visual analysis, Abstract, p. 1).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Zhang so that the entity is a porcine or bovine animal because Zin suggests that this is needed by farmers who have cows on their dairy farm, to efficiently monitor the cows (1. Introduction, p. 1-2).
Claim(s) 12-14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kruse (US 20230384114A1) in view of Zhang (see citation below).
As per Claim 12, Claim 12 is similar in scope to Claim 1. Kruse does not teach wherein the machine learning module comprises a set of code stored in the memory and when executed by the processor cause the machine learning module to: generate or select a set of simulated fiducial tags; generate a set of transformed fiducial tags by randomly transforming each fiducial tag from the set of simulated fiducial tags; generate a set of superimposed training images by superimposing each transformed fiducial tag from the set of transformed fiducial tags on a random image from a set of images; and train a convolutional neural network by processing each superimposed training image in the set of superimposed training images. However, Zhang teaches these limitations, as discussed in the rejection for Claim 1. Thus, Claim 12 is rejected under the same rationale as Claim 1.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Kruse so that the machine learning module comprises a set of code stored in the memory and when executed by the processor cause the machine learning module to: generate or select a set of simulated fiducial tags; generate a set of transformed fiducial tags by randomly transforming each fiducial tag from the set of simulated fiducial tags; generate a set of superimposed training images by superimposing each transformed fiducial tag from the set of transformed fiducial tags on a random image from a set of images; and train a convolutional neural network by processing each superimposed training image in the set of superimposed training images because Zhang suggests that this achieves consistently better detection robustness and higher pose accuracy (p. 12).
As per Claim 13, Kruse teaches wherein the fiducial identification module comprises a neural network configured to identify the set of fiducial tags associated with the set of entities (neural network, [0123], [0034]).
However, Kruse does not expressly teach that the neural network is a convolutional neural network. However, Zhang teaches wherein the fiducial identification module comprises a convolutional neural network configured to identify the set of fiducial tags (CNN-based fiducial marker detector, p. 3, left column, last sentence).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Kruse so that the neural network is a convolutional neural network as suggested by Zhang. It is well-known in the art that CNN has the advantages of automatic feature extraction, leading to high accuracy in tasks like image recognition; parameter sharing, which reduces computational cost and model size; and invariance to transformations, meaning they can recognize objects regardless of their position or orientation in an image.
As per Claim 14, Kruse teaches wherein the system further comprises an entity identification module; and wherein the entity identification module comprises a set of code stored in the memory and when executed by the processor cause the entity identification module to: associate a set of characters on the fiducial tag (21) with an entity (fiducial markers 21, fiducial marker 21 indicating one or more landmark features in environment, the one or more landmark features may include an object, fiducial marker 21 includes encoded fiducial data, fiducial data may include a code embodied on fiducial marker 21, fiducial marker 21 may include a machine-readable 2-dimensional pattern, such as barcodes, QR codes, [0043], [0268]).
Claim(s) 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kruse (US 20230384114A1) and Zhang (see citation below) in view of Csaszar (US 20130076522A1).
Claim 16 is similar in scope to Claim 2, and therefore is rejected under the same rationale.
Claim(s) 17-18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kruse (US 20230384114A1) and Zhang (see citation below) in view of Rice (see citation below).
Claims 17-18 are similar in scope to Claims 3-4 respectively, and therefore are rejected under the same rationale.
Claim(s) 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kruse (US 20230384114A1) and Zhang (see citation below) in view of Gu (see citation below).
Claim 19 is similar in scope to Claim 5, and therefore is rejected under the same rationale.
Claim(s) 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kruse (US 20230384114A1), Zhang (see citation below), and Gu (see citation below) in view of Li (US 20210365716A1).
Claim 20 is similar in scope to Claim 6, and therefore is rejected under the same rationale.
Claim(s) 21 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kruse (US 20230384114A1), Zhang (see citation below), Gu (see citation below), and Li (US 20210365716A1) in view of Buttler (see citation below).
Claim 21 is similar in scope to Claim 7, and therefore is rejected under the same rationale.
Prior Art of Record
1. Zhuming Zhang et al; DeepTag: A General Framework for Fiducial Marker Design and Detection; May 2021; IEEE Transactions on Pattern Analysis and Machine Intelligence; p. 1-12; https://arxiv.org/pdf/2105.13731v1
2. Andrew Rice et al; Dependable Coding of Fiducial Tags; November 2004; Proceedings of the Second International Conference on Ubiquitous Computing Systems; p. 259-274; https://link.springer.com/chapter/10.1007/11526858_20
3. Wenhao Gu et al; Extended Capture Range of Rigid 2D/3D Registration by Estimating Riemannian Pose Gradients; September 2020; Lecture Notes in Computer Science; volume 12436; p. 281-289; https://link.springer.com/chapter/10.1007/978-3-030-59861-7_29
4. Carmen Buttler et al; Single molecule fate of HIV-1 envelope reveals late-stage viral lattice incorporation; May 2018; Nature Communications; p. 1-13; https://www.nature.com/articles/s41467-018-04220-w
5. Thi Thi Zin et al; Automatic Cow Location Tracking System Using Ear Tag Visual Analysis; June 2020; Sensors; vol. 20; no. 12; p. 1-16; https://pmc.ncbi.nlm.nih.gov/articles/PMC7349613/pdf/sensors-20-03564.pdf
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JONI HSU whose telephone number is (571)272-7785. The examiner can normally be reached M-F 10am-6:30pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee Tung can be reached at (571)272-7794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
JH
/JONI HSU/Primary Examiner, Art Unit 2611