EcoliWiki:Gene Association File

This page is undergoing a major revision. Please be aware that this page may change suddenly.

EcoliWiki and EcoCyc contribute to the Gene Ontology (GO) by monthly submitting our annotations.

Format

The gene_assocation file format has two versions. The Gene Ontology's documentation page describes these quite well. Below are some notes, some specific to the EcoCyc/EcoliWiki version of the gaf.

Column Number	Name	Notes	Cardinality
1	Database	This is the database that the identifiers come from. We are using EcoCyc identifiers, so this column will always be "EcoCyc."	1
2	db_object_id	Unique identifier from the database in column1. We are using the EcoCyc database identifiers for historical reasons. These accessions are pulled out of the 'Gene_accessions_table' on the Gene page. These EcoCyc accessions must be up to date or the entire set of annotations for a particular protein can be lost.	1
3	db_object_symbol	A unique and valid symbol. This is the canonical name of the subject of the annotation (a gene or protein name, ORF, etc.) In this case it is the name of the gene product. We use the standard E. coli systematic nomenclature and capitolize the first letter of this symbol to make it clear this is a product. This is where a lot of collision between EcoliWiki and EcoCyc happens. After the two files of correct annotations from the two databases are combined, the names in this column usually conflict with each other -- meaning there is more than one db_object_symbol for a particular db_object_id.	1
4	Qualifier	The qualifier. These three are nonstandard and must be removed: obsolote_go_term - flag under_review - we think this should be recurated deprecated - we've looked at it and it needs to be removed	0, 1, >1 for cardinality >1 use a pipe to separate entries (e.g. NOT|contributes_to)
5	GO id		1
6	Reference	A valid database prefix is required	1
7	Evidnece	Three letter abbreviation	1
8	With/From	column of variable content, depending on the situation. See the Gene Ontology gaf2.0 specification page for more info.	0, 1
9	Aspect	one of three: P, F, C	1
10	db_object_name	name of the gene or gene product. free text	0, 1
11	db_object_synonym	a pipe-delimited list of synonyms for this subject. Comes from the synonyms field in the 'Product_nomenclature_table'.	0, 1, >1
12	db_object_type	A description of the type of gene product being annotated. If column 17 is used, this column refers to that, otherwise, it refers to column 2	1
13	taxon	integer value of the taxon. It is not quite clear which taxon we are annotating to. See. EcoliWiki:Gene Association File#Taxon Controversy	1
14	date	Date on which the annotation was made. IEAs older than a year should be removed.	1
15	assigned by	the database which made the annotation. Not the person. One of the values in the GO list of valid database identifiers.	1
16	annotation extension	contains cross-references to other ontologies that can be used to qualify or enhance the annotation. prefaced by an appropriate GO relationship	0, 1, >1
17	isoform/product form	as the db_object_id (column 2) must be a canonical entry - something that has a 1:1 relationship to a gene - this field allows the annotation of "specific variants of that gene or gene product." EcoliWiki as of July 2010 has no way of annotating to a specific variant of a gene product. This column must contain a standard DB:ID identifier.	?

Generation

See text for description.

The two sites' annotations are prepared by the EcoliWiki team (mostly Daniel,) and submitted via CVS to the GO Consortium. The figure at right shows the four basic steps to this process:

1. Obtain annotations from EcoCyc (quarterly)
2. Obtain annotations from EcoliWiki (monthly)
3. Merge them and resolve name-conflicts
4. Submit to the GO

Taxon Identifier(s)

• EcoliWiki does not offer the option to annotate to any other taxon besides taxon:83333 (Escherichia coli K-12).
• EcoCyc annotates to taxon:511145 (Escherichia coli str. K-12 substr. MG1655) regardless of the strain/species used in the experiment.

This is less than optimal, as important information about the strain is not captured by the annotations from either database.

Non-standard Qualifiers in EcoliWiki

EcoliWiki expands upon the idea of the qualifier to include the following:

obsolote_go_term
under_review
deprecated

There are actually two files that get submitted each month. The gene association file and the gp2protein file. A detailed description of each can be found in the GO Consortium's documentation.

Step 0.5 - Setup the environment

The process requires some files and scripts from the GO Consortium. Instead of do this by hand each month, Daniel wrote a script to handle this part of the process. To use this bash script, copy this code into a file and make it executable. On our servers it is named gaf_setup.sh.

The first step is to run this script. Alternatively, you can follow the directions below to do this by hand.

gaf_setup.sh

<php>

1. !/bin/bash
2. -------------------------------------------------------------------------------
4. gaf_setup.sh Daniel Renfro 01/2010
6. This script sets up a working directory for the montly creation of the
7. gene-associations file and the gp2protein file for EcoliWiki.
9. -------------------------------------------------------------------------------

working_dir=$(date "+%Y-%m-%d") prefix=`dirname "$0"`

1. make the working directory

mkdir $working_dir cd $working_dir

1. make some necessary directory

echo "Creating directories..." mkdir ontology mkdir doc

1. download the stuff from GO

echo "Downloading files from the Gene Ontology..." ncftp ftp://ftp.geneontology.org/pub/go/software/utilities/ <<END_SCRIPT get filter-gene-association.pl get check-abbr-ga-file.pl quit END_SCRIPT ncftp ftp://ftp.geneontology.org/pub/go/ontology <<END_SCRIPT get gene_ontology_edit.obo quit END_SCRIPT ncftp ftp://ftp.geneontology.org/pub/go/doc/ <<END_SCRIPT get GO.xrf_abbs quit END_SCRIPT

1. put the files in the right places

mv gene_ontology_edit.obo ontology mv GO.xrf_abbs doc

1. set the permissions on the executable files.

chmod +x check-abbr-ga-file.pl filter-gene-association.pl

echo "setup complete."</php>

manual setup

Make a new directory to work in.

cd /usr/local/working/GAF/
mkdir YYYY-MM-DD
chmod 775 YYYY-MM-DD
cd YYYY-MM-DD

Look through the wiki and find things that are wrong and can be fixed and fix them.

php5 /usr/local/phpwikibots/trunk/find_GO_terms_with_changed_names.php -w /Library/WebServer/WebServer/Documents/ecoliwiki/colipedia 
php5 /usr/local/phpwikibots/trunk/find_obsolete_categories_and_annotations.php -w /Library/WebServer/WebServer/Documents/ecoliwiki/ -C -A 

Make some directories for use later

mkdir {doc, ontology}

Download some scripts and some files. Here we are using the ncftp commandline tool, you can use whichever tool you are comfortable with.

ncftp ftp://ftp.geneontology.org/pub/go/software/utilities/
  get filter-gene-association.pl
  get check-abbr-ga-file.pl
  quit
ncftp ftp://ftp.geneontology.org/pub/go/ontology
  get gene_ontology_edit.obo
  quit
ncftp ftp://ftp.geneontology.org/pub/go/doc/
  get GO.xrf_abbs
  quit

Move some of those files into the directories.

mv gene_ontology_edit.obo ontology/
mv GO.xrf_abbs doc/

Step 1 - EcoCyc annotation

EcoCyc's quarterly releases contain their annotations in the gene_association file format. The file is typically named gene_associations.ecocyc.... (Notice the plural on associations). In order to prepare their annotations, we must first parse the annotations for validity. Using the script(s) we downloaded in the setup section, parse the most recent EcoCyc release's annotation file. For a more detailed description of the script, see http://www.geneontology.org/GO.format.annotation.shtml#script.

./filter-gene-association.pl -i gene_associations.ecocyc-13.0 -p nocheck -e 

This script will print the erroneous lines with short descriptions of the problem. Some of these errors will be easy to fix and will not have any biological significance. A common error is misspellings or disallowed entries in the Assigned_by column (column 15.) Other errors are non-trivial and should be kept track of for further curation.

I need to make sure this information gets back to EcoCyc.
--Daniel 22:43, 8 March 2010 (UTC)

Get the good lines from the EcoCyc release, for later inclusion into the GAF.

./filter-gene-association.pl -i gene_association.ecocyc-13.0 -p nocheck -w > good_lines.ecocyc

Step 2 - EcoliWiki annotations

Get the annotations from EcoliWiki. We use a custom-written script to pull annotations from the wiki, named mine_annotations.php.

php5 mine_annotations.php -w ecoliwiki/colipedia > gene_associations.ecoliwiki 

Just like with the EcoCyc annotations, we must validate the annotations. The lack of restrictions in the annotation table makes for some interesting errors, most of which can be fixed by hand. These errors need to be fixed in the wiki as well, or they will show up again next month.

./filter-gene-association.pl -i gene_associations.ecoliwiki -p nocheck -e > bad_lines.ecoliwiki

Using a text editor, the errors are fixed. Some sample errors are:

a.) unprintable character found on this line, may not be this column
b.) Line identical to previous line
c.) Qualifier column=4 allowed type not present, found "Deprecated"
d.) DB:Reference column=6 format of reference not DB:REFID

Some characters, such as 'β', cannot be included as there is no standard encoding for the GAF and they can become garbled. Also, EcoliWiki does not enforce strict reference notation, so some lines may need a PubMed number. EcoliWiki also uses some non-standard qualifiers such as deprecated and under_review. These are internal qualifiers meant to tag annotations for later curation. To get a list of these, run the following command (requires ack).

ack -A1 "column=4" bad_lines.ecoliwiki | ack -v '^[-\d]' > other_qualifiers.ecoliwiki

I need to also make sure that these annotations are curated and/or sent to EcoCyc.

After the file has been fixed to an acceptable degree, the good lines are obtained for inclusion into the GAF.

./filter-gene-association.pl -i gene_associations.ecoliwiki -p nocheck -w > good_lines.ecoliwiki

Step 3 - Merge

In some cases the names of the products are different in the two websites. This requires the manual changing of some fields in the file. This is definitely the rate-limiting step.

First, some background:

• To facilitate inter-website information exchange EcoliWiki uses the EcoCyc internal naming scheme to refer to the annotations.
• For example, in the DnaJ accessions table there is a link to EcoCyc's DnaJ page (EcoCyc:EG10240-MONOMER.)
• This is the internal EcoCyc name for that product. EcoliWiki thus puts EG10240-MONOMER in column 2 in its gene_associations.ecoliwiki file. This is to make merging easier.
• Column 2 in the GAF spec is db_object_id -- the internal name used.
• Column 3 in the GAF spec is db_object_symbol -- this can be thought of as the common-name of the product, in this case: DnaJ.

Problems arise when EcoliWiki and EcoCyc use different db_object_symbols/common-names for the same db_object_id. (Column 2 matches, but column 3 doesn't.) The GO filter script counts this as an error.

To successfully merge the two websites' files, we first concatenate them together.

cat good_lines.ecocyc good_lines.ecoliwiki > combined_good_lines.gaf

The next step is to run the filter script, save the output and fix the file by hand.

./filter-gene-association.pl -i combined_good_lines.gaf -p nocheck -e > name_conflicts.txt

Using the name_conflicts.txt file, we can find all the occurrences where naming differences resulted in errors. Below is a table listing some of the common name-conflicts encountered. Where there was a discrepancy between EcoliWiki and EcoCyc, the name was looked up in EcoGene and in the literature.

Make a full list of things I've come across and send to Markus at EcoCyc.
--Daniel 23:39, 8 March 2010 (UTC)

DB_object_ID	EcoCyc id	EcoliWiki id	EcoGene id	what I did in this case...
LACTALDDEHYDROG-MONOMER	Ald	AldA	AldA	changed Ald to AldA
G8209-MONOMER	MolR_3	molR_3 (molR)	yehH'	MolR_3
TRANSKETOI-MONOMER	Tkt	TktA	TktA	changed to TktA
THREDEHYDCAT-MONOMER	Tdc	TdcB	TdcB	TdcB
G7743-MONOMER	Hsp15	HslR	HslR	HslR That's an L, not a 1
G7744-MONOMER	Hsp33	HslO	HslO	HslO
EG10600-MONOMER	GroE	GroES	---	GroES based on what is in the description and literature
EG10599-MONOMER	---	GroEL product	---	removed the "product"
EG12859-MONOMER	MdoD	OpgD	OpgD	OpgD
EG10427	HemA	GtrA	HemA	HemA
G7158-MONOMER	YfaU	RhmA	RhmA	RhmA

The last editing step before the submissions is to remove the IEA annotations that are older than 1 year.

Step 4 - Submit

header information

Occasionally the header information in the gene_association file needs to be updated. This information can be found in the gene_assocation.ecocyc.conf file in the go/gene-associations/submission/ directory of the GO CVS. To change this information, just edit the file from the CVS checkout and resubmit it. It will get prepended to the gene_assocation file automatically. As of March 2010 the file looks like this:

project_name=EcoCyc and EcoliWiki collaborative annotation for E. coli K-12
contact_email=biocyc-support@ai.sri.com
contact_email=ecoliwiki@gmail.com
project_url=http://ecoliwiki.net
project_url=http://ecocyc.org
funding_source=GM077678
funding_source=NIGMS U24GM088849
email_report=ecoliwiki@gmail.com
ecocyc_version=14.0

gp2protein.ecocyc

Aside from the gene_association file, the gp2protein file needs to be submitted. This can be made on-the-fly each month from a custom script that pulls the necessary information from EcoliWiki.

submission via CVS

The files () are sorted and uniqed using the appropriate Unix commands. They are then gzipped and moved into the CVS directory.

cd ~/CVS/
CVS submit

See Also

• How EcoliWiki incorporates EcoCyc's annotations
• GO Consortium's specification for the GAF - http://www.geneontology.org/GO.format.annotation.shtml
• gp2protein file notes - http://wiki.geneontology.org/index.php/Gp2protein