Build a local database

GeneSpy uses custom local databases, which means you are in charge of building your own !

The generation of the database is composed of 3 steps:

• Downloading GFF files

• Formatting the GFF database

• Creating list of strains

You can use the 3 in 1 module (GeneSpy>Database>Basic Database) or the Advanced construction (GeneSpy>Database>Advanced …).

Select and load a database

Finally, you should tell GeneSpy where to look for the GFM files and the list of strains (Automatically done for 3 in 1 module).

1. Got to Database>Select GFM database… and select the GFM directory.
2. Go to Database>Select strains list… and select the file containing the strains list.

• Paths are saved, so you do not have to do this operation until you change your database.

• Decoupling of GFM directory and strain list allows to link a GFM database to many lists (different sampling, different displayed names, …).

• The strains list can be viewed and edited in Menu>Edit>Strains list…

• The state of the database is indicated in the box “Database”. This box displays:

  • The number of GFM files in the selected folder

  • The number of strains in the list file

  • The number of genomes that match between GFM collection and strains list (efficient genomes)

Clicking on this box displays the list of assembly numbers and their associated state. (Orphan GFM files, orphan strains in list or overlaping genomes). Here is an example of a database presenting all possibilities:

Using this list, you can identify potential problems in your database. Only overlapping genomes can be used by GeneSpy.

• When the database is selected and ready, go to Load data section!

Combine multiple databases/add new genomes

User can fuse many databases or add new genomes to a pre-existent database.

• To combine several databases from the same source:

  1. Retrieve collections of GFF (and assembly reports) in a same folder A.

  2. Put all old GFM in the database directory B.

  3. Perform Format, selecting A and B (New GFM and new SQL database will be created).

  4. Create new strains list, selecting A and B.

  5. Give paths to GeneSpy.

• To combine several databases from NCBI and other sources (e.g. RAST):

  1. Build databases independently in different repertories (GFF: directory A and B, GFM directory A’ and B’).

  2. Put all GFM in the database directory C.

  3. Format the database, selecting any folder (A or B) and C (Only SQL database will be created).

  4. Concatenate all strains lists in a same file.

  5. Give paths to GeneSpy.

Edit strains list

Sometime, default strains names does not fit with data/format. You can easily change strains names that are displayed on figures or that are exported in the iTOL format.

Go to Menu>Edit>Strains list…

Two panels are displayed. The first corresponds to the current list of strains and the second one corresponds to the sub-selection of strains.

• To edit names of strains:

  1. click on the desired strain.

  2. Edit the name in the entry at the bottom of the list.

  3. Click on “Edit strain” and answer “yes”.

NB: the line has to be in format <Assembly>\t<Name of strain>\t<Other name>, else, the new name is not accepted.

• To delete a strain:

  1. click on the desired strain.

  2. Click on “Delete strain” and answer “yes”.

Create a subdatabase

A subdatabase can be created, filtering genomes with the strains list.

Go to Menu>Edit>Strains list…

Two panels are displayed. The first corresponds to the current list of strains and the second one corresponds to the sub-selection of strains.

1. Build the sub-selection:

• Add a strain : Double click on any strain/click on any strain and click on “Add to selection”.

• Delete a strain : Press “Suppr”/click on any strain and click on “Delete”.

• To perform an automatic sampling (one strain per species) : click on “Auto sampling”.

2. Save the sub-selection : click on “Save”.

3. Import the new selection : Menu>Database>Select strains list.. and select the new file.

From a file of target genes

You can manually define a set of genes of interest in a tabular file and import it in GeneSpy.

Select Files>Load… and select your file (.txt)

NB : You can also load a file and export it directly without displaying the figure on interface (Files>Load and export…). By this way, there is no limit of number of loaded contexts (The only limitation is the height and width proportion of the final image).

From a keyword search

You have the possibility to search any gene in your database using the Search menu.

1. In the field “Species/part of strains”, you can choose genomes to query.
   • If you want to query entire database, keep this field empty.
   • If you want to query specific genomes, type the name of strain/species you want (or just a part of the name).
     Example : “bacillus” queries all genomes containing “bacillus” in the name (e.g. Bacillus subtilis, Bacillus thuringiensis).
     Example : “entero” queries all genomes containing “entero” in the name (e.g. Enterococcus faecalis, Enterobacter cloacae).
     

2. In the field “keywords”, you can choose the keyword that will be searched in genomes. Example : “ribosomal” queries all genes containing “ribosomal” in their description (e.g. ribosomal protein S4, ribosomal protein 14).

3. Click on “search from keywords”/Press Enter to launch the search.

4. Click on “Add to selection”.

5. Click on “Load selection”.

Importantly, the search is not case sensitive.

From a NCBI BLASTP result

User can also find target genes from a simple NCBI BLASTP result file.

1. Ensure that you possess a NCBI GeneSpy database.

2. Go to https://blast.ncbi.nlm.nih.gov/Blast.cgi?PAGE=Proteins.

3. Choose a sequence of interest that you want to analyze.

4. Blast your sequence against a taxon corresponding to GeneSpy database (BLASTP).

Example : I’v got a GeneSpy database of Lactobacillus casei built from NCBI. I perform the NCBI BLASTP against Lactobacillus casei (field “Organism”).

5. Select All.

6. Download XML.

7. Menu>Search>from NCBI BLASTP output.

8. Select the database that you used to build your GeneSpy’s database (RefSeq/GenBank).

9. Select the XML file.

GeneSpy retrieves all accession numbers present in NCBI output file and tries to find corresponding accession numbers in GFM files. If some genes are found, they will appear in the Search window text box.

NB: At the end of the list of matches, the number of hits that have been found is indicated.

10. Click on “Add to selection”.

11. Click on “Load selection”.

From a local BLASTP result

User can also find target genes from a local BLASTP result file.

1. Ensure that you possess a protein sequence local database (FAA files) corresponding exactly to your GeneSpy database (GFF and GFM files). Accession numbers have to be the same in both databases.(See Advanced section to build local database).

2. Choose a sequence of interest that you want to analyze.

3. Perform a local BLASTP with output default parameters.

4. Ensure that accession numbers are present in output file.

5. Menu>Search>from local BLASTP output.

6. Type regular expression to fit with output file format: up and downstream of accession number, all regular expression has to cover the entire field “Sequences producing significant alignments”.

• example 1:

  Sequences producing significant alignments: (bits) Value

  170187-000006885.1-Spneumoniae-@WP_050545454.1@[hypothetical … 836 0.0

  The regular expression is « .+@» for upstream and «@.+» for downstream of accession number

• example 2:

  Sequences producing significant alignments: (bits) Value

  170187-121155511511-PROTEIN~WP_00000045.1 836 0.0

  The regular expression is « .+\~» for upstream and «$» for downstream of accession number

7. Choose an e-value threshold (ex: 1e-2 or 0.01 ; default 1e-4).

8. Select your BLASTP output file.

NB: At the end of the list of matches, the number of hits that have been found is indicated.

9. Click on “Add to selection”.

10. Click on “Load selection”.

From a GenBank collection

User can also load target genes from a collection of GenBank files (.gb and .gp). GeneSpy extracts all identifiers contained in GenBank files located in a given folder and searches in its database if some of them are indexed. Importantly, all genes present in GenBank files are searched, so if the GenBank file contains a whole genomic region, all genes will be searched! We suggest using GenBank files corresponding to individual genes.

1. Ensure that you possess a NCBI GeneSpy database.

2. Go to Menu>Search>From GenBank collection.

3. Select the folder containing GenBank files.

NB: At the end of the list of matches, the number of hits that have been found is indicated.

4. Click on “Add to selection”.

5. Click on “Load selection”.

From a list of accession numbers

User can load target genes from a file containing a list of acession numbers (1 accession number per line).

1. Go to Menu>Search>From accession list file.

2. Select the file containing accession numbers.

NB: At the end of the list of matches, the number of hits that have been found is indicated.

3. Click on “Add to selection”.

4. Click on “Load selection”.

From a selection

When you interact with the interface, you can select specific genes clicking on them.

• Mouse wheel click/Double left click selects any gene.
• Right click selects target gene.

The selection will be displayed in selection box in the form <Assembly><Accession>. The selection can be loaded clicking on “Load selection”. It displays selected genes.

Left click on genes displays important information about the given gene in the gene information box. Click on the box open a NCBI web page of the gene (works only if database comes from NCBI).

Order of contexts can be changed in a selection: use copy/paste to change the order in the selection box and load selection.

General options

Many options are available to customize your figures:

Custom colors

Images

GeneSpy offers a variety of export formats (depending on OS):

• png
• jpg
• tif
• svg
• eps
• pdf
• multiple pdf

iTOL

Using GeneSpy, you can directly bind genomic context information to a phylogeny. For this, you have to export your context as a text file that can be directly loaded in iTOL ([http://itol.embl.de/]).

Names of leaves in your tree have to be exactly the same as in the iTOL text file. That’s why different formats are proposed:

• Assembly
• Assembly_Accession
• Other name
• Other name_Accession

If default names generated by GeneSpy do not fit with your data, you can edit names manually in Edit>List of strains… or create a script that changes the list of strains. For that purpose, see Advanced data section.

Use the Convert_IDs function

GeneSpy provides the possibility to load IDs in a different format as the default format. This can be very conveniant if your data are not formatted as GeneSpy identifiers and to avoid repetitive file conversions. This possibility is provided by the function Convert_IDs in GeneSpy_tools.py script.

For example, I want to copy/paste leaves from a tree that are in this format: 484770_000725345_1_Psp_@WP_038667413_1. The aim is that GeneSpy understands directly this format and interprets it as GCA_000725345.1 WP_038667413.1.

Many conditions are required to use this function:

• User has to have two databases: the first one containing original names (protein database, nucleic database, identifiers database, …) and a compatible GeneSpy’s database.

• Assembly and accession numbers have to be included in the original names.

• The formats are different (different separators, different order, ..).

The function takes a string as argument (a string corresponding to a line that has been given in a file or in selection box) and returns a string containing the assembly number and the accession number. Importantly, the separator in the final string is “#”. The function has to be edited to fit with your data.

Example 1:

I have an original protein database built from NCBI RefSeq and a GeneSpy’s database corresponding to the same genomes. I have inferred a tree from an alignment of sequences from this database. I want to load directly leaves from the tree (e.g. Copy/Paste from Figtree) in GeneSpy in the format:

484770_000725345_1_Psp_@WP_038667413_1
171540_000925743_1_Bsubtilis_@$WP_038456466_1

Original line : 484770_000725345_1_Psp_@WP_038667413_1

def Convert_IDs(string_id):
    string_id_genespy_format = ""
    try:
      "# this code is the adaptative part that has to be edited according to your needs #"
      accession_version = string_id.split("@")[1].split("_")
      version = accession_version.pop()
      accession = "_".join(accession_version)
      assembly = "GCA_"+string_id.split("_")[1]+"."+string_id.split("_")[2]
      string_id_genespy_format = assembly+"#"+accession+"."+version
      " #######################"
      return string_id_genespy_format
    except:
      return string_id
  

Formatted line : GCA_000725345.1#WP_038667413.1

Example 2:

I have an original protein database built from NCBI RefSeq and a GeneSpy’s database corresponding to the same genomes. I have retrieved sequences in a fasta file. I want to load directly the fasta file in GeneSpy in the format:

>GCA_000725345.1$4550012$WP_038667413.1
ATGGTTTGTAGCGATCGGGCTATCGAGTCTATCGGGCGCATAT
>GCA_000925743.1$171540$WP_038456466.1
ATGTAGCGGCTTAGGCCTCTTCGGAGGCTCTAGCGGGTATCTA

Original line : >GCA_000725345.1$4550012$WP_038667413.1

def Convert_IDs(string_id):
    string_id_genespy_format = ""
    try:
      "# this code is the adaptative part that has to be edited according to your needs #"
      accession_version = string_id.split("$")[2]
      assembly = string_id.split("$")[0].replace(">","")
      string_id_genespy_format = assembly+"#"+accession
      " #######################"
      return string_id_genespy_format
    except:
      return string_id
  

Formatted line : GCA_000725345.1#WP_038667413.1

NB : regular identifiers <Assembly> <Accession> can be also understood evenif the function is active. Convert_IDs just provides an alternative way to read identifiers in case the identifier is not readable.

Proxy

Edition of links

Links are sometime not optimal. In this example, the surrounded gene is FtsQ but is not annotated as such.

• That’s why GeneSpy provides a way to edit links between gene names and functions:

1. Go to Menu>Edit>Color/Binding…

2. Enter the name and the annotation in respective fields and click on “change data”.

GeneSpy first tests if annotation exists and then links the annotation to name. Any gene that has the annotation will display the name and the associated color. If the gene name did not exist, the default color is red.

3. Refresh

If you want to erase a link between a name and a function, you can link the function to a “_” in name field. An inferred family is displayed in a dull color, this can be changed in Options>Coloration>Equal.

• Colors can also be changed:

1. Enter the name or the annotation and double click on associated color entry.

2. Select the desire color and click on “change data”.

3. Refresh

• You also can get a color of any gene name or function:

Keyboard shortcut

• Open a file : Ctrl+o

• Save as : Ctrl+s

• Quit : Ctrl+q

• Refresh : F5

Formats

• Format of GFM files:
  #Contig (l.1)
  <inf>\t<sup>\t<orientation>\t<accession>\t<locus tag>\t<annotation>\t[<name>]

• Format of NCBI FTP links file:
  “<RefSeq FTP>”,“<GenBank FTP>”

• Format of List of strains:
  <Assembly>\t<Name of strain>\t<Other name (displayed in iTOL formats, useful to fit with phylogenies)>

More generally, assembly corresponds to the name of the GFM file.

• Format of GFF files:
  <Contig><source><region/gene/CDS><start><end><score><strand><phase><attributes>
  attributes: Name=<Accession number> ; product=<Biochemical function> ; locus_tag=<locus tag> ; genome/type=<genomic/plasmidic> ; gene=<name of gene> ; ID=<Accession number> (RAST/PROKKA)

• Format of Target file:
  <Assembly>\t<Accession>

• Format of NCBI BLASTP output (line of interest only):
  xml (l.1)
  blastp (l.5)
  [<Hit_id>,<Hit_def>][ref,dbj,gb]|<Accession>[</Hit_id>,</Hit_def>]

• Format of local BLASTP output (line of interest only):
  BLASTP (l.1)
  <Hit (default: <Accession><Annotation>)> <Score> <e-value>

• Format of list of accession numbers:
  <Accession number 1>
  <Accession number 2>
  <Accession number 3>

• Format of GenBank file (line of interest only):
  LOCUS (l.1)
  /protein_id=“<Accession>”

• Format of GenPept file (line of interest only):
  LOCUS (l.1)
  ACCESSION <Accession>

• Format of color file with annotation:
  ><color>
  <Annotation (accession/locus tag/function/name)>

• Format of color file with identifiers:
  ><color>
  >Assembly>\t<Accession>

• Format of iTOL specific format:
  DATASET_DOMAINS
  SEPARATOR COMMA
  DATASET_LABEL,Genomic Context
  DATA
  <Leaf>,<windows lengh>,[TL,TR]|<begin>|<end>|<color>|<name>

• Format of SQL database:
  Creation: IDS (<assembly>, <accession>)
  Query: SELECT assembly FROM IDS WHERE accession=<Accession>

Loading data in GeneSpy

Classical way:

1. Extract content, check the format.

2. Check if ABC genome is in the list of strains.

3. Check if ABC.gfm exists.

4. Search 123 gene in ABC.gfm.

5. Generate the context.

Search menu (from file):

1. Extract content, check the format, extract accession numbers.

2. Search GFM files associated to this accession number in SQL database.

3. Contruct of the conventionnal ID.

4. Check if ABC genome is in the list of strains.

5. Check if ABC.gfm exists.

6. Search 123 gene in ABC.gfm.

7. Generate the context.

Databases

Limits

• Maximum number of contexts that can be displayed on interface : 330.

• Maximum number of contexts that can be exported in images using “Load and export” : depending on the width and height of final image.

• Maximum number of contexts that can be exported in iTOL using “Load and export” : no limit.

Known issues

• Problem of installation:
  If you have a SSL certificate issue during the installation of libraries, you can try to type in a terminal (Unix) "curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py" and then "python2.7 get-pip.py".

• Problem of memory leak:
  Using matplotlib and Tkinter causes a memory leak that is partially fixed sometimes. You can change memory threshold in PATHS_PARAMS.ini to minimize the number of restarts. Don’t panic if the window closes, all data are saved!

• Problem in MacOS:
  Due to interaction between Tkinter and Matplotlib, GeneSpy can be very slow to clean dispayed figures when refreshing/loading new data and when scrolling in the frame containing figure.

• Problem in Windows:
  Due to interaction between Tkinter and Matplotlib, GeneSpy can be very slow to clean dispayed figures when refreshing/loading new data.