biomaRt/DESCRIPTION0000644000175400017540000000367513214603701014713 0ustar00biocbuildbiocbuildPackage: biomaRt Version: 2.34.1 Title: Interface to BioMart databases (e.g. Ensembl, COSMIC, Wormbase and Gramene) Authors@R: c(person("Steffen", "Durinck", role = c("aut"), email = "biomartdev@gmail.com"), person("Wolfgang", "Huber", role="aut"), person("Sean", "Davis", role="ctb", email = "sdavis2@mail.nih.gov"), person("Francois", "Pepin", role="ctb"), person(given = "Vince S", family = "Buffalo", role="ctb"), person("Mike", "Smith", role=c("ctb", "cre"), email = "grimbough@gmail.com")) Depends: methods Imports: utils, XML, RCurl, AnnotationDbi, progress, stringr, httr Suggests: annotate, BiocStyle, knitr, rmarkdown, testthat VignetteBuilder: knitr biocViews: Annotation Description: In recent years a wealth of biological data has become available in public data repositories. Easy access to these valuable data resources and firm integration with data analysis is needed for comprehensive bioinformatics data analysis. biomaRt provides an interface to a growing collection of databases implementing the BioMart software suite (http://www.biomart.org). The package enables retrieval of large amounts of data in a uniform way without the need to know the underlying database schemas or write complex SQL queries. Examples of BioMart databases are Ensembl, COSMIC, Uniprot, HGNC, Gramene, Wormbase and dbSNP mapped to Ensembl. These major databases give biomaRt users direct access to a diverse set of data and enable a wide range of powerful online queries from gene annotation to database mining. License: Artistic-2.0 LazyLoad: yes NeedsCompilation: no Packaged: 2017-12-14 23:22:41 UTC; biocbuild Author: Steffen Durinck [aut], Wolfgang Huber [aut], Sean Davis [ctb], Francois Pepin [ctb], Vince S Buffalo [ctb], Mike Smith [ctb, cre] Maintainer: Mike Smith biomaRt/NAMESPACE0000644000175400017540000000133513214572320014415 0ustar00biocbuildbiocbuildimport(methods) import(RCurl,XML) importFrom(utils, edit, head, read.table) importFrom(AnnotationDbi, keys, columns, keytypes, select) importFrom(progress, progress_bar) importFrom(stringr, str_extract_all, str_match) importFrom(httr, POST, content) #for some reason RCurl needs to have findHTTPHeaderEncoding exported - #remove it from the exports if this ever gets fixed export(listMarts, getGene, getSequence, exportFASTA, useMart, listDatasets, useDataset, listEnsembl, useEnsembl, listAttributes, listFilters, getBM, getXML,getLDS, attributePages, filterOptions,filterType, getBMlist, NP2009code, keys, columns, keytypes, select, listEnsemblArchives) exportClasses(Mart) exportMethods("show") biomaRt/NEWS0000644000175400017540000000206613175713423013706 0ustar00biocbuildbiocbuildCHANGES IN VERSION 2.34.0 ------------------------- NEW FEATURES o Added the listEnsemblArchives() function. This returns a table of the available Ensembl archives, and replaces the archive = TRUE argument to several functions, which was no longer working. BUG FIXES o The Ensembl BioMart server doesn't always respond well if queries with more than 500 filter values are submitted. If a query that exceed this is detect biomaRt will now submit the query in batches and concatonate the result when completed. MINOR CHANGES o You can now provide a host with 'http://' at the start, or a trailing '/' (typically copy/pasted from a browser) and useMarts() will cope. CHANGES IN VERSION 2.32.0 ------------------------- BUG FIXES o Fixed bug when columns were not returned in the order requested, which resulted in the wrong column names being added to the result. CHANGES IN VERSION 2.30.0 ------------------------- SIGNIFICANT USER-LEVEL CHANGES o Updated vignette to use BiocStyle and execute most code chunks. biomaRt/R/0000755000175400017540000000000013214572320013375 5ustar00biocbuildbiocbuildbiomaRt/R/biomaRt.R0000644000175400017540000013511513214572320015123 0ustar00biocbuildbiocbuild ########################## #biomaRt source code # ########################## # # #Licence: Artistic # #Author: Steffen Durinck # ########################## ############### #messageToUser# ############### messageToUser <- function(message){ if(interactive()){ cat(message) } } ############################################################## #martCheck # # # #This function checks if there is a valid Mart object, # #if a dataset is selected and # #if the correct BioMart database has been selected (optional)# ############################################################## martCheck = function(mart, biomart = NULL){ if( missing( mart ) || class( mart ) != 'Mart'){ stop("You must provide a valid Mart object. To create a Mart object use the function: useMart. Check ?useMart for more information.") } if(!is.null(biomart)){ # martcheck = strsplit(martBM(mart),"_", fixed = TRUE, useBytes = TRUE)[[1]][1] martcheck = martBM(mart) bmok = FALSE for(k in 1:length(biomart)){ if(martcheck[1] == biomart[k]){ bmok = TRUE } } if(!bmok){ stop(paste("This function only works when used with the ",biomart," BioMart.",sep="")) } } if(martDataset(mart)==""){ stop("No dataset selected, please select a dataset first. You can see the available datasets by using the listDatasets function see ?listDatasets for more information. Then you should create the Mart object by using the useMart function. See ?useMart for more information"); } } checkWrapperArgs = function(id, type, mart){ if(missing(type))stop("Specify the type of identifier you are using, see ?getGene for details. Valid values for the type argument can be found with the listFilters function.") if(!type %in% listFilters(mart)[,1])stop(paste("Invalid identifier type:",type," see ?getGene for details. Use the listFilters function to get the valid value for the type argument.", sep="")) if(missing(id))stop("No identifiers specified. Use the id argument to specify a vector of identifiers for which you want to retrieve the annotation.") } bmRequest <- function(request, ssl.verifypeer = TRUE, verbose = FALSE){ if(verbose) writeLines(paste("Attempting web service request:\n",request, sep="")) result = tryCatch(getURL(request, ssl.verifypeer = ssl.verifypeer, followlocation = TRUE), error = function(e){ message("Request to BioMart web service failed.\n", "The BioMart web service you're accessing may be down.\n", "Check the following URL and see if this website is available:\n", request) }) return(result) } ####################################################### #listMarts: # #list all available BioMart databases by default # #listMarts will check the central service to see which# #BioMart databases are present # ####################################################### listMarts <- function( mart = NULL, host="www.ensembl.org", path="/biomart/martservice", port=80, includeHosts = FALSE, archive = FALSE, ssl.verifypeer = TRUE, ensemblRedirect = TRUE, verbose = FALSE){ request = NULL if(is.null(mart)){ ## adding option to force use of specificed host with ensembl redirect <- ifelse(!ensemblRedirect && grepl(x = host, pattern = "ensembl.org"), "&redirect=no", "") host <- .cleanHostURL(host) if(archive) { warning("The archive = TRUE argument is now deprecated.\nUse listEnsemblMarts() to find the URL to directly query an Ensembl archive.") request = paste0(host, ":", port, path, "?type=registry_archive&requestid=biomaRt") } else { request = paste0(host, ":", port, path, "?type=registry&requestid=biomaRt", redirect) } } else{ if(class(mart) == 'Mart'){ request = paste0(martHost(mart), "?type=registry&requestid=biomaRt") } else{ warning(paste(mart,"object needs to be of class Mart created with the useMart function. If you don't have a Mart object yet, use listMarts without arguments or only specify the host argument",sep=" ")) } } registry = bmRequest(request = request, ssl.verifypeer = ssl.verifypeer, verbose = verbose) ## check this looks like the MartRegistry XML, otherwise throw an error if(!grepl(x = registry, pattern = "^\n*")) { stop('Unexpected format to the list of available marts.\n', 'Please check the following URL manually, ', 'and try ?listMarts for advice.\n', request) } registry = xmlTreeParse(registry, asText=TRUE) registry = registry$doc$children[[1]] marts = list(biomart = NULL, version = NULL, host = NULL, path = NULL, database = NULL) index = 1 if(host != "www.biomart.org" || archive){ for(i in seq(len=xmlSize(registry))){ if(xmlName(registry[[i]])=="MartURLLocation"){ if(xmlGetAttr(registry[[i]],"visible") == 1){ if(!is.null(xmlGetAttr(registry[[i]],"name"))) marts$biomart[index] = as.character(xmlGetAttr(registry[[i]],"name")) if(!is.null(xmlGetAttr(registry[[i]],"database"))) marts$database[index] = as.character(xmlGetAttr(registry[[i]],"database")) if(!is.null(xmlGetAttr(registry[[i]],"displayName"))) marts$version[index] = as.character(xmlGetAttr(registry[[i]],"displayName")) if(!is.null(xmlGetAttr(registry[[i]],"host"))) marts$host[index] = as.character(xmlGetAttr(registry[[i]],"host")) if(!is.null(xmlGetAttr(registry[[i]],"path"))) marts$path[index] = as.character(xmlGetAttr(registry[[i]],"path")) if(!is.null(xmlGetAttr(registry[[i]],"port"))) marts$port[index] = as.character(xmlGetAttr(registry[[i]],"port")) if(!is.null(xmlGetAttr(registry[[i]],"serverVirtualSchema"))){ marts$vschema[index] = as.character(xmlGetAttr(registry[[i]],"serverVirtualSchema")) } index=index+1 } } } } else{ for(i in seq(len=xmlSize(registry))){ if(xmlName(registry[[i]])=="MartURLLocation"){ if(xmlGetAttr(registry[[i]],"visible") == 1){ if(!is.null(xmlGetAttr(registry[[i]],"name"))) marts$biomart[index] = xmlGetAttr(registry[[i]],"name") if(!is.null(xmlGetAttr(registry[[i]],"database"))) marts$database[index] = xmlGetAttr(registry[[i]],"database") if(!is.null(xmlGetAttr(registry[[i]],"displayName"))) marts$version[index] = xmlGetAttr(registry[[i]],"displayName") marts$host[index] = host marts$path[index] = path marts$port[index] = 80 if(!is.null(xmlGetAttr(registry[[i]],"serverVirtualSchema"))){ marts$vschema[index] = xmlGetAttr(registry[[i]],"serverVirtualSchema") } index=index+1 } } } } if(includeHosts){ return(marts) } else{ if(archive){ ret = data.frame(biomart = as.character(marts$database),version = as.character(marts$version), stringsAsFactors=FALSE) } else{ ret = data.frame(biomart = as.character(marts$biomart),version = as.character(marts$version), stringsAsFactors=FALSE) } return(ret) } } ################################# # # # # # # Generic BioMart functions # # # # # # ################################# useMart <- function(biomart, dataset, host = "www.ensembl.org", path = "/biomart/martservice", port = 80, archive = FALSE, ssl.verifypeer = TRUE, ensemblRedirect = TRUE, version, verbose = FALSE){ if(missing(biomart) && missing(version)) stop("No biomart databases specified. Specify a biomart database to use using the biomart or version argument") if(!missing(biomart)){ if(!(is.character(biomart))) stop("biomart argument is not a string. ", "The biomart argument should be a single character string") } if(biomart == "ensembl" & grepl(x = host, pattern = "ensembl.org")) { biomart = "ENSEMBL_MART_ENSEMBL" } reqHost = host host <- .cleanHostURL(host) marts <- listMarts(host=host, path=path, port=port, includeHosts = TRUE, archive = archive, ssl.verifypeer = ssl.verifypeer, ensemblRedirect = ensemblRedirect) mindex = NA if(!missing(biomart)){ mindex=match(biomart,marts$biomart) } if(!missing(version)){ mindex=match(version,marts$version) } if(is.na(mindex) || archive){ mindex=match(biomart,marts$database) } if(is.na(mindex)) stop("Incorrect BioMart name, use the listMarts function to see which BioMart databases are available") if(is.na(marts$path[mindex]) || is.na(marts$vschema[mindex]) || is.na(marts$host[mindex]) || is.na(marts$port[mindex]) || is.na(marts$path[mindex])) stop("The selected biomart databases is not available due to error in the BioMart central registry, please report so the BioMart registry file can be fixed.") if(marts$path[mindex]=="") marts$path[mindex]="/biomart/martservice" #temporary to catch bugs in registry if(archive) biomart = marts$biomart[mindex] if(!missing(version)) biomart = marts$biomart[mindex] biomart = sub(" ","%20",biomart, fixed = TRUE, useBytes = TRUE) ## adding option to force use of specificed host with ensembl redirect <- ifelse(!ensemblRedirect && grepl(x = host, pattern = "ensembl.org"), "?redirect=no", "") mart <- new("Mart", biomart = biomart, vschema = marts$vschema[mindex], host = paste0(host, ":", marts$port[mindex], marts$path[mindex], redirect), archive = archive) if(length(grep("archive",martHost(mart)) > 0)){ if(length(grep(reqHost,martHost(mart))) == 0){ writeLines(paste("Note: requested host was redirected from ", reqHost, " to " ,martHost(mart),sep="")) writeLines("When using archived Ensembl versions this sometimes can result in connecting to a newer version than the intended Ensembl version") writeLines("Check your ensembl version using listMarts(mart)") } } BioMartVersion=bmVersion(mart, verbose=verbose) #### below can probably be deleted, MS - 05/04/2017 #if(martHost(mart) =="http://www.biomart.org:80/biomart/martservice"){ # if(verbose) writeLines("Using Central Repository at www.biomart.org"); # martVSchema(mart) <- 'default' #Assume central service query uses default vSchema #} #### if(verbose){ writeLines(paste("BioMartServer running BioMart version:",BioMartVersion,sep=" ")) writeLines(paste("Mart virtual schema:",martVSchema(mart),sep=" ")) if(length(grep(reqHost,martHost(mart))) == 0){ writeLines(paste("Requested host was redirected from ", reqHost, " to " ,martHost(mart),sep="")) } writeLines(paste("Mart host:",martHost(mart),sep=" ")) } if(!missing(dataset)){ mart = useDataset(mart = mart, dataset=dataset, verbose = verbose) } return(mart) } listDatasets <- function(mart, verbose = FALSE) { if(missing(mart) || !is(mart, 'Mart')) stop("No Mart object given or object not of class 'Mart'") ## we choose a separator based on whether 'redirect=no' is present sep <- ifelse(grepl(x = martHost(mart), pattern = ".+\\?.+"), "&", "?") request = paste0(martHost(mart), sep, "type=datasets&requestid=biomaRt&mart=", martBM(mart)) bmResult = bmRequest(request = request, verbose = verbose) con = textConnection(bmResult) txt = scan(con, sep="\t", blank.lines.skip=TRUE, what="character", quiet=TRUE, quote = "\"") #txt = tryCatch(scan(request, sep="\t", blank.lines.skip=TRUE, what="character", quiet=TRUE), error = function(e){stop("Request to BioMart web service failed. Verify if you are still connected to the internet. Alternatively the BioMart web service is temporarily down.")}) close(con) ## select visible ("1") table sets i = intersect(which(txt=="TableSet"), which(txt=="1")-3L) res = data.frame(dataset = I(txt[i+1L]), description = I(txt[i+2L]), version = I(txt[i+4L])) return(res) } ## Check version of BioMart service bmVersion <- function(mart, verbose=FALSE){ ## we choose a separator based on whether 'redirect=no' is present sep <- ifelse(grepl(x = martHost(mart), pattern = ".+\\?.+"), "&", "?") request = paste0(martHost(mart), sep, "type=version", "&requestid=biomaRt&mart=", martBM(mart)) BioMartVersion = bmRequest(request = request, verbose = verbose) bmv = "" if(BioMartVersion == "\n" | BioMartVersion == ""){ bmv = NA if(verbose) warning(paste("BioMart version is not available from BioMart server:",request,sep="\n")) } else{ con = textConnection(BioMartVersion) bmVersionParsed = read.table(con, sep="\t", header=FALSE, quote = "", comment.char = "", as.is=TRUE) close(con) if(verbose) print(bmVersionParsed) if(dim(bmVersionParsed)[2] >= 1){ bmv=bmVersionParsed[1,1] } } return(bmv) } ## Retrieve attributes and filters from web service bmAttrFilt <- function(type, mart, verbose=FALSE){ ## we choose a separator based on whether 'redirect=no' is present sep <- ifelse(grepl(x = martHost(mart), pattern = ".+\\?.+"), "&", "?") request = paste0(martHost(mart), sep, "type=", type, "&dataset=", martDataset(mart), "&requestid=biomaRt&mart=", martBM(mart), "&virtualSchema=", martVSchema(mart)) attrfilt = bmRequest(request = request, verbose = verbose) con = textConnection(attrfilt) attrfiltParsed = read.table(con, sep="\t", header=FALSE, quote = "", comment.char = "", as.is=TRUE) close(con) if(type=="attributes"){ if(dim(attrfiltParsed)[2] < 3) stop("biomaRt error: looks like we're connecting to incompatible version of BioMart suite.") cnames = seq_len(dim(attrfiltParsed)[2]) cnames=paste(type,cnames,sep="") cnames[1] = "name" cnames[2] = "description" cnames[3] = "fullDescription" if(dim(attrfiltParsed)[2] < 4){ warning("biomaRt warning: looks like we're connecting to an older ", "version of BioMart suite.\nSome biomaRt functions might ", "not work.") } else{ cnames[4] = "page" } colnames(attrfiltParsed) = cnames } if(type=="filters"){ if(dim(attrfiltParsed)[2] < 4) stop("biomaRt error: looks like we're connecting to incompatible version of BioMart suite.") cnames = seq(1:dim(attrfiltParsed)[2]) cnames=paste(type,cnames,sep="") cnames[1] = "name" cnames[2] = "description" cnames[3] = "options" cnames[4] = "fullDescription" if(dim(attrfiltParsed)[2] < 7){ warning("biomaRt warning: looks like we're connecting to an older version of BioMart suite. Some biomaRt functions might not work.") } else{ cnames[5] = "filters" cnames[6] = "type" cnames[7] = "operation" } colnames(attrfiltParsed) = cnames } return(attrfiltParsed) } ## Utilty function to check dataset specification ## Returns dataset name as a character assuming all checks ## have been passed. checkDataset <- function(dataset, mart) { validDatasets=listDatasets(mart) ## subseting data.frames can produce some weird classes ## which aren't character(), so we coerce it here dataset <- as.character(dataset) if(length(dataset) > 1) stop("Please only specify a single dataset name") if(is.na(match(dataset, validDatasets$dataset))) stop(paste("The given dataset: ",dataset,", is not valid. Correct dataset names can be obtained with the listDatasets() function.")) return(dataset) } ## Select a BioMart dataset useDataset <- function(dataset, mart, verbose = FALSE){ if(missing(mart) || class(mart)!="Mart") stop("No valid Mart object given, specify a Mart object with the attribute mart") if(missing(dataset)) { stop("No dataset given. Please use the dataset argument to specify which dataset you want to use. Correct dataset names can be obtained with the listDatasets function.") } else { dataset <- checkDataset(dataset = dataset, mart = mart) } martDataset(mart) <- dataset if(verbose) messageToUser("Checking attributes ...") martAttributes(mart) <- bmAttrFilt("attributes",mart, verbose = verbose) if(verbose){ messageToUser(" ok\n") messageToUser("Checking filters ...") } martFilters(mart) <- bmAttrFilt("filters",mart, verbose = verbose) if(verbose) messageToUser(" ok\n") return( mart ) } ## getName getName <- function(x, pos) if(is.null(x[[pos]])) NA else x[[pos]] ## listAttributes listAttributes <- function(mart, page, what = c("name","description","page")) { martCheck(mart) if(!missing(page) && !page %in% attributePages(mart)) stop(paste("The chosen page: ",page," is not valid, please use the correct page name using the attributePages function",sep="")) attrib=NULL if(!missing(page)){ sel = which(martAttributes(mart)[,"page"] == page) attrib = martAttributes(mart)[sel,what] } else{ attrib = martAttributes(mart)[,what] } return(attrib) } ## attributePages attributePages <- function(mart){ martCheck(mart) pages = unique(martAttributes(mart)[,"page"]) return(pages) } ## listFilters listFilters <- function(mart, what = c("name", "description")) { martCheck(mart) filters = martFilters(mart) badwhat = !(what %in% colnames(filters)) if(any(badwhat)) stop(sprintf("The function argument 'what' contains %s: %s\nValid are: %s\n", if(sum(badwhat)>1) "invalid values" else "an invalid value", paste(what[badwhat], collapse=", "), paste(colnames(filters), collapse=", "))) return(filters[, what]) } ## filterOptions filterOptions <- function(filter, mart){ if(missing(filter)) stop("No filter given. Please specify the filter for which you want to retrieve the possible values.") if(class(filter)!="character")stop("Filter argument should be of class character") martCheck(mart) if(!filter %in% listFilters(mart, what="name"))stop("Filter not valid, check for typo in filter argument.") sel = which(listFilters(mart, what="name") == filter) return(listFilters(mart,what="options")[sel]) } ## filterType filterType <- function(filter, mart){ if(missing(filter)) stop("No filter given. Please specify the filter for which you want to retrieve the filter type") if(class(filter)!="character")stop("Filter argument should be of class character") martCheck(mart) type="unknown" sel = which(listFilters(mart, what="name") == filter) if(is.null(sel))stop(paste("Invalid filter",filter, sep=": ")) type = listFilters(mart,what="type")[sel] return(type) } ########################################## #getBM: generic BioMart query function # ########################################## getBM <- function(attributes, filters = "", values = "", mart, curl = NULL, checkFilters = TRUE, verbose=FALSE, uniqueRows=TRUE, bmHeader=FALSE, quote="\""){ martCheck(mart) if(missing( attributes )) stop("Argument 'attributes' must be specified.") if(is.list(filters) && !missing( values )) warning("Argument 'values' should not be used when argument 'filters' is a list and will be ignored.") if(is.list(filters) && is.null(names(filters))) stop("Argument 'filters' must be a named list when sent as a list.") if(!is.list(filters) && filters != "" && missing( values )) stop("Argument 'values' must be specified.") if(length(filters) > 0 && length(values) == 0) stop("Values argument contains no data.") if(is.list(filters)){ values = filters filters = names(filters) } if(class(uniqueRows) != "logical") stop("Argument 'uniqueRows' must be a logical value, so either TRUE or FALSE") ## force the query to return the 'english text' header names with the result ## we use these later to match and order attribute/column names callHeader <- TRUE xmlQuery = paste0(" ") #checking the Attributes invalid = !(attributes %in% listAttributes(mart, what="name")) if(any(invalid)) stop(paste("Invalid attribute(s):", paste(attributes[invalid], collapse=", "), "\nPlease use the function 'listAttributes' to get valid attribute names")) #check if attributes come from multiple attribute pages currently disabled until ID issue resovled at Ensembl if(FALSE){ att = listAttributes(mart, what=c("name","page")) att = att[which(att[,1] %in% attributes),] attOK = FALSE pages = unique(att[,2]) if(length(pages) <= 1){ attOK = TRUE } else{ for(page in pages){ if(length(attributes) == length(which(attributes %in% att[which(att[,2] == page),1]))) attOK = TRUE } } if(!attOK){ stop(paste("Querying attributes from multiple attribute pages is not allowed. To see the attribute pages attributes belong to, use the function attributePages.")) } } #attribute are ok lets add them to the query attributeXML = paste("", collapse="", sep="") #checking the filters if(filters[1] != "" && checkFilters){ invalid = !(filters %in% listFilters(mart, what="name")) if(any(invalid)) stop(paste("Invalid filters(s):", paste(filters[invalid], collapse=", "), "\nPlease use the function 'listFilters' to get valid filter names")) } ## filterXML is a list containing filters with reduced numbers of values ## to meet the 500 value limit in BioMart queries filterXmlList <- .generateFilterXML(filters, values, mart) resultList <- list() if(length(filterXmlList) > 1) { pb <- progress_bar$new(total = length(filterXmlList), width = options()$width - 10, format = "Batch submitting query [:bar] :percent eta: :eta") pb$tick(0) } ## we submit a query for each chunk of the filter list for(i in seq_along(filterXmlList)) { if(exists('pb')) { pb$tick() } filterXML <- filterXmlList[[ i ]] fullXmlQuery = paste(xmlQuery, attributeXML, filterXML,"",sep="") if(verbose) { message(fullXmlQuery) } ## we choose a separator based on whether '?redirect=no' is present sep <- ifelse(grepl(x = martHost(mart), pattern = ".+\\?.+"), "&", "?") ## postRes = tryCatch(postForm(paste0(martHost(mart), sep),"query" = fullXmlQuery), ## error = function(e) { ## stop("Request to BioMart web service failed. Verify if you are still connected to the internet. Alternatively the BioMart web service is temporarily down.") ## } ## ) postRes <- .submitQuery(host = paste0(martHost(mart), sep), query = fullXmlQuery) if(verbose){ writeLines("#################\nResults from server:") print(postRes) } if(!(is.character(postRes) && (length(postRes)==1L))) stop("The query to the BioMart webservice returned an invalid result: biomaRt expected a character string of length 1. Please report this to the mailing list.") if(gsub("\n", "", postRes, fixed = TRUE, useBytes = TRUE) == "") { # meaning an empty result result = as.data.frame(matrix("", ncol=length(attributes), nrow=0), stringsAsFactors=FALSE) } else { if(length(grep("^Query ERROR", postRes))>0L) stop(postRes) ## convert the serialized table into a dataframe con = textConnection(postRes) result = read.table(con, sep="\t", header=callHeader, quote = quote, comment.char = "", check.names = FALSE, stringsAsFactors=FALSE) if(verbose){ writeLines("#################\nParsed results:") print(result) } close(con) if(!(is(result, "data.frame") && (ncol(result)==length(attributes)))) { print(head(result)) stop("The query to the BioMart webservice returned an invalid result: the number of columns in the result table does not equal the number of attributes in the query. Please report this to the mailing list.") } } resultList[[i]] <- .setResultColNames(result, mart = mart, attributes = attributes, bmHeader = bmHeader) } ## collate results result <- do.call('rbind', resultList) return(result) } ################################### #getLDS: Multiple dataset linking # ################################### getLDS <- function(attributes, filters = "", values = "", mart, attributesL, filtersL = "", valuesL = "", martL, verbose = FALSE, uniqueRows = TRUE, bmHeader = TRUE) { martCheck(mart) martCheck(martL) invalid = !(attributes %in% listAttributes(mart, what="name")) if(any(invalid)) stop(paste("Invalid attribute(s):", paste(attributes[invalid], collapse=", "), "\nPlease use the function 'listAttributes' to get valid attribute names")) invalid = !(attributesL %in% listAttributes(martL, what="name")) if(any(invalid)) stop(paste("Invalid attribute(s):", paste(attributesL[invalid], collapse=", "), "\nPlease use the function 'listAttributes' to get valid attribute names")) if(filters[1] != ""){ invalid = !(filters %in% listFilters(mart, what="name")) if(any(invalid)) stop(paste("Invalid filters(s):", paste(filters[invalid], collapse=", "), "\nPlease use the function 'listFilters' to get valid filter names")) } if(filtersL[1] != ""){ invalid = !(filtersL %in% listFilters(martL, what="name")) if(any(invalid)) stop(paste("Invalid filters(s):", paste(filtersL[invalid], collapse=", "), "\nPlease use the function 'listFilters' to get valid filter names")) } xmlQuery = paste(" ",sep="") attributeXML = paste("", collapse="", sep="") if(length(filters) > 1){ if(class(values)!= "list") stop("If using multiple filters, the 'value' has to be a list.\nFor example, a valid list for 'value' could be: list(affyid=c('1939_at','1000_at'), chromosome= '16')\nHere we select on affyid and chromosome, only results that pass both filters will be returned"); filterXML = NULL for(i in seq(along=filters)){ if(filterType(filters[i],mart) == 'boolean' || filterType(filters[i],mart) == 'boolean_list'){ if(!is.logical(values[[i]])) stop(paste("biomaRt error: ",filters[i]," is a boolean filter and needs a corresponding logical value of TRUE or FALSE to indicate if the query should retrieve all data that fulfill the boolean or alternatively that all data that not fulfill the requirement should be retrieved."), sep="") if(!values[[i]]){ values[[i]] = 1 } else{ values[[i]] = 0 } filterXML = paste(filterXML,paste("", collapse="",sep=""),sep="") } else{ valuesString = paste(values[[i]],"",collapse=",",sep="") filterXML = paste(filterXML,paste("", collapse="",sep=""),sep="") } } } else{ if(filters != ""){ if(filterType(filters,mart) == 'boolean' || filterType(filters,mart) == 'boolean_list'){ if(!is.logical(values)) stop(paste("biomaRt error: ",filters," is a boolean filter and needs a corresponding logical value of TRUE or FALSE to indicate if the query should retrieve all data that fulfill the boolean or alternatively that all data that not fulfill the requirement should be retrieved."), sep="") if(!values){ values = 1 } else{ values = 0 } filterXML = paste("", collapse="",sep="") } else{ valuesString = paste(values,"",collapse=",",sep="") filterXML = paste("", collapse="",sep="") } } else{ filterXML="" } } xmlQuery = paste(xmlQuery, attributeXML, filterXML,"",sep="") xmlQuery = paste(xmlQuery, "", sep="") linkedAttributeXML = paste("", collapse="", sep="") if(length(filtersL) > 1){ if(class(valuesL)!= "list") stop("If using multiple filters, the 'value' has to be a list.\nFor example, a valid list for 'value' could be: list(affyid=c('1939_at','1000_at'), chromosome= '16')\nHere we select on affyid and chromosome, only results that pass both filters will be returned"); linkedFilterXML = NULL for(i in seq(along=filtersL)){ if(filterType(filtersL,martL) == 'boolean' || filterType(filtersL,martL) == 'boolean_list'){ if(!is.logical(valuesL[[i]])) stop(paste("biomaRt error: ",filtersL[i]," is a boolean filter and needs a corresponding logical value of TRUE or FALSE to indicate if the query should retrieve all data that fulfill the boolean or alternatively that all data that not fulfill the requirement should be retrieved."), sep="") if(!valuesL[[i]]){ valuesL[[i]] = 1 } else{ valuesL[[i]] = 0 } linkedFilterXML = paste(linkedFilterXML,paste("", collapse="",sep=""),sep="") } else{ valuesString = paste(valuesL[[i]],"",collapse=",",sep="") linkedFilterXML = paste(linkedFilterXML,paste("", collapse="",sep=""),sep="") } } } else{ if(filtersL != ""){ if(filterType(filtersL,martL) == 'boolean' || filterType(filtersL,martL) == 'boolean_list'){ if(!is.logical(valuesL)) stop(paste("biomaRt error: ",filtersL," is a boolean filter and needs a corresponding logical value of TRUE or FALSE to indicate if the query should retrieve all data that fulfill the boolean or alternatively that all data that not fulfill the requirement should be retrieved."), sep="") if(!valuesL){ valuesL = 1 } else{ valuesL = 0 } linkedFilterXML = paste("", collapse="",sep="") } else{ valuesString = paste(valuesL,"",collapse=",",sep="") linkedFilterXML = paste("", collapse="",sep="") } } else{ linkedFilterXML="" } } xmlQuery = paste(xmlQuery, linkedAttributeXML, linkedFilterXML,"",sep="") if(verbose){ cat(paste(xmlQuery,"\n", sep="")) } #postRes = postForm(paste(martHost(mart),"?",sep=""),"query"=xmlQuery) ## we choose a separator based on whether '?redirect=no' is present sep <- ifelse(grepl(x = martHost(mart), pattern = ".+\\?.+"), "&", "?") ## POST query postRes <- .submitQuery(host = paste0(martHost(mart), sep), query = xmlQuery) ## 10-01-2014 if(length(grep("^Query ERROR", postRes))>0L) stop(postRes) ## if(postRes != ""){ con = textConnection(postRes) result = read.table(con, sep="\t", header=bmHeader, quote = "\"", comment.char = "", as.is=TRUE, check.names = TRUE) close(con) if(nrow(result) > 0 && all(is.na(result[,ncol(result)]))) result = result[,-ncol(result),drop=FALSE] ## 10 - 01 - 2014 res_attributes <- c(attributes,attributesL) if(!(is(result, "data.frame") && (ncol(result)==length(res_attributes)))) { print(head(result)) stop("The query to the BioMart webservice returned an invalid result: the number of columns in the result table does not equal the number of attributes in the query. Please report this to the mailing list.") } if(!bmHeader){ #assumes order of results same as order of attibutes in input colnames(result) = res_attributes ## } } else { warning("getLDS returns NULL.") result=NULL } return(result) } ###################### #getXML ###################### getXML <- function(host="http://www.biomart.org/biomart/martservice?", xmlquery){ pf = postForm(host,"query"=xmlquery) con = textConnection(pf) result = read.table(con, sep="\t", header=FALSE, quote = "", comment.char = "", as.is=TRUE) close(con) return(result) } ###################### #getBMlist ###################### getBMlist <- function(attributes, filters = "", values = "", mart, list.names = NULL, na.value = NA, verbose=FALSE, giveWarning=TRUE){ if(giveWarning) writeLines("Performing your query using getBM is preferred as getBMlist perfoms a separate getBM query for each of the values one gives. This is ok for a short list but will definitely fail when used with longer lists. Ideally one does a batch query with getBM and then iterates over that result.") out <- vector("list", length(attributes)) if(is.null(list.names)) names(out) <- attributes else names(out) <- list.names for(j in seq(along = attributes)){ tmp2 <- vector("list", length(values)) names(tmp2) <- values for(k in seq(along = tmp2)){ tst <- getBM(attributes = attributes[j], filters=filters, values = values[k], mart = mart, verbose = verbose) if(class(tst) == "data.frame"){ tmp <- unlist(unique(tst[!is.na(tst)]), use.names = FALSE) if(length(tmp) > 0) tmp2[[k]] <- tmp else tmp2[[k]] <- na.value }else{ tmp2[[k]] <- na.value } out[[j]] <- tmp2 } } return(out) } ############################### # # #Ensembl specific functions # ############################### listEnsembl <- function(mart = NULL, host="www.ensembl.org",version = NULL, GRCh = NULL, mirror = NULL,verbose = FALSE){ if(!is.null(mirror) & (!is.null(version) | !is.null(GRCh))){ warning("version or GRCh arguments can not be used together with the mirror argument. Will ignore the mirror argument and connect to default ensembl host") mirror = NULL } if(!is.null(version)){ host = paste("e",version,".ensembl.org",sep="") } if(!is.null(GRCh)){ if(GRCh == 37){ host = paste("grch",GRCh,".ensembl.org",sep="") } else{ print("Only 37 can be specified for GRCh version") } } ensemblRedirect <- TRUE if(!is.null(mirror)){ if(!(mirror %in% c("www", "uswest", "useast", "asia"))) { warning("Invalid mirror select a mirror from [www, uswest, useast, asia].\n", "default when no mirror is specified is to be redirected to the ", "closest mirror to your location") } else { host <- paste0(mirror, ".ensembl.org") ensemblRedirect <- FALSE } } marts = listMarts(mart = mart, host = host, verbose = verbose, ensemblRedirect = ensemblRedirect) sel = which(marts$biomart == "ENSEMBL_MART_ENSEMBL") if(length(sel) > 0){ marts$biomart[sel] = "ensembl" } sel = which(marts$biomart == "ENSEMBL_MART_SNP") if(length(sel) > 0){ marts$biomart[sel] = "snp" } sel = which(marts$biomart == "ENSEMBL_MART_FUNCGEN") if(length(sel) > 0){ marts$biomart[sel] = "regulation" } sel = which(marts$biomart == "ENSEMBL_MART_VEGA") if(length(sel) > 0){ marts$biomart[sel] = "vega" } return(marts) } useEnsembl <- function(biomart, dataset,host = "www.ensembl.org", version = NULL, GRCh = NULL, mirror = NULL, verbose = FALSE){ if(!is.null(mirror) & (!is.null(version) | !is.null(GRCh))){ warning("version or GRCh arguments can not be used together with the mirror argument. Will ignore the mirror argument and connect to default ensembl host") mirror = NULL } if(!is.null(version)){ host = paste("e",version,".ensembl.org",sep="") } if(!is.null(GRCh)){ if(GRCh == 37){ host = paste("grch",GRCh,".ensembl.org",sep="") } else{ print("Only 37 can be specified for GRCh version") } } ensemblRedirect <- TRUE if(!is.null(mirror)){ if(!(mirror %in% c("www", "uswest", "useast", "asia"))) { warning("Invalid mirror select a mirror from [www, uswest, useast, asia].\n", "default when no mirror is specified is to be redirected to the ", "closest mirror to your location") } else { host <- paste0(mirror, ".ensembl.org") ensemblRedirect <- FALSE } } if(biomart == "ensembl"){ biomart = "ENSEMBL_MART_ENSEMBL" } if(biomart == "snp"){ biomart = "ENSEMBL_MART_SNP" } if(biomart == "regulation"){ biomart = "ENSEMBL_MART_FUNCGEN" } if(biomart == "vega"){ biomart = "ENSEMBL_MART_VEGA" } ens = useMart(biomart = biomart, dataset = dataset, host = host, verbose = verbose, ensemblRedirect = ensemblRedirect) return(ens) } getGene <- function( id, type, mart){ martCheck(mart,"ensembl") checkWrapperArgs(id, type, mart) symbolAttrib = switch(strsplit(martDataset(mart), "_", fixed = TRUE, useBytes = TRUE)[[1]][1],hsapiens = "hgnc_symbol",mmusculus = "mgi_symbol","external_gene_id") typeAttrib = switch(type,affy_hg_u133a_2 = "affy_hg_u133a_v2",type) attrib = c(typeAttrib,symbolAttrib,"description","chromosome_name","band","strand","start_position","end_position","ensembl_gene_id") table = getBM(attributes = attrib,filters = type, values = id, mart=mart) return(table) } getSequence <- function(chromosome, start, end, id, type, seqType, upstream, downstream, mart, verbose=FALSE){ martCheck(mart,c("ensembl","ENSEMBL_MART_ENSEMBL")) if(missing(seqType) || !seqType %in% c("cdna","peptide","3utr","5utr", "gene_exon", "transcript_exon","transcript_exon_intron","gene_exon_intron","coding","coding_transcript_flank","coding_gene_flank","transcript_flank","gene_flank")){ stop("Please specify the type of sequence that needs to be retrieved when using biomaRt in web service mode. Choose either gene_exon, transcript_exon,transcript_exon_intron, gene_exon_intron, cdna, coding,coding_transcript_flank,coding_gene_flank,transcript_flank,gene_flank,peptide, 3utr or 5utr") } if(missing(type))stop("Please specify the type argument. If you use chromosomal coordinates to retrieve sequences, then the type argument will specify the type of gene indentifiers that you will retrieve with the sequences. If you use a vector of identifiers to retrieve the sequences, the type argument specifies the type of identifiers you are using.") if(missing(id) && missing(chromosome) && !missing(type))stop("No vector of identifiers given. Please use the id argument to give a vector of identifiers for which you want to retrieve the sequences.") if(!missing(chromosome) && !missing(id))stop("The getSequence function retrieves sequences given a vector of identifiers specified with the id argument of a type specified by the type argument. Or alternatively getSequence retrieves sequences given a chromosome, a start and a stop position on the chromosome. As you specified both a vector of identifiers and chromsomal coordinates. Your query won't be processed.") if(!missing(chromosome)){ if(!missing(start) && missing(end))stop("You specified a chromosomal start position but no end position. Please also specify a chromosomal end position.") if(!missing(end) && missing(start))stop("You specified a chromosomal end position but no start position. Please also specify a chromosomal start position.") if(!missing(start)){ start = as.integer(start) end = as.integer(end) } if(missing(upstream) && missing(downstream)){ sequence = getBM(c(seqType,type), filters = c("chromosome_name","start","end"), values = list(chromosome, start, end), mart = mart, checkFilters = FALSE, verbose=verbose) } else{ if(!missing(upstream) && missing(downstream)){ sequence = getBM(c(seqType,type), filters = c("chromosome_name","start","end","upstream_flank"), values = list(chromosome, start, end, upstream), mart = mart, checkFilters = FALSE, verbose=verbose) } if(!missing(downstream) && missing(upstream)){ sequence = getBM(c(seqType,type), filters = c("chromosome_name","start","end","downstream_flank"), values = list(chromosome, start, end, downstream), mart = mart, checkFilters = FALSE, verbose = verbose) } if(!missing(downstream) && !missing(upstream)){ stop("Currently getSequence only allows the user to specify either an upstream of a downstream argument but not both.") } } } if(!missing(id)){ if(missing(type)) stop("Type argument is missing. This will be used to retrieve an identifier along with the sequence so one knows which gene it is from. Use the listFilters function to select a valid type argument.") if(!type %in% listFilters(mart, what="name")) stop("Invalid type argument. Use the listFilters function to select a valid type argument.") valuesString = paste(id,"",collapse=",",sep="") if(missing(upstream) && missing(downstream)){ sequence = getBM(c(seqType,type), filters = type, values = id, mart = mart, verbose=verbose) } else{ if(!missing(upstream) && missing(downstream)){ sequence = getBM(c(seqType,type), filters = c(type, "upstream_flank"), values = list(id, upstream), mart = mart, checkFilters = FALSE, verbose=verbose) } if(!missing(downstream) && missing(upstream)){ sequence = getBM(c(seqType,type), filters = c(type, "downstream_flank"), values = list(id, downstream), mart = mart, checkFilters = FALSE, verbose=verbose) } if(!missing(downstream) && !missing(upstream)){ stop("Currently getSequence only allows the user to specify either an upstream of a downstream argument but not both.") } } } return(sequence) } #################### #export FASTA # #################### exportFASTA <- function( sequences, file ){ if( missing( sequences ) || class( sequences ) != "data.frame"){ stop("No data.frame given to write FASTA. The data.frame should be the output of the getSequence function."); } if( missing(file)){ stop("Please provide filename to write to"); } if(length(sequences[1,]) == 2){ for(i in seq(along = sequences[,2])){ cat(paste(">",sequences[i,2],"\n",sep=""),file = file, append=TRUE); cat(as.character(sequences[i,1]),file = file, append = TRUE); cat("\n\n", file = file, append = TRUE); } } else{ for(i in seq(along = sequences[,2])){ cat(paste(">chromosome_",sequences[i,1],"_start_",sequences[i,2],"_end_",sequences[i,3],"\n",sep=""),file = file, append=TRUE); cat(as.character(sequences[i,4]),file = file, append = TRUE); cat("\n\n", file = file, append = TRUE); } } } ################### #Nature Protocol ################### NP2009code <- function(){ edit(file=system.file('scripts', 'Integration-NP.R', package = 'biomaRt')) }biomaRt/R/biomaRtClasses.R0000644000175400017540000000105713175713423016445 0ustar00biocbuildbiocbuildsetClass("Mart", representation(biomart = "character", host = "character", vschema = "character", version = "character", dataset = "character", filters = "data.frame", attributes = "data.frame", archive = "logical" ), prototype(dataset = "", vschema="default", archive = FALSE ) ); biomaRt/R/ensembl.R0000644000175400017540000000154313214572320015150 0ustar00biocbuildbiocbuild## location of Ensembl specific functions ## scrapes the ensembl website for the list of current archives and returns ## a data frame containing the versions and their URL listEnsemblArchives <- function() { html <- htmlParse("http://www.ensembl.org/info/website/archives/index.html?redirect=no") archive_box <- getNodeSet(html, path = "//div[@class='plain-box float-right archive-box']")[[1]] archive_box_string <- toString.XMLNode(archive_box) archives <- strsplit(archive_box_string, split = "
  • ")[[1]][-1] extracted <- str_extract_all(string = archives, pattern = "Ensembl [A-Za-z0-9 ]{2,6}|http://.*ensembl\\.org|[A-Z][a-z]{2} [0-9]{4}") tab <- do.call("rbind", extracted) colnames(tab) <- c("url", "version", "date") tab <- tab[,c(2,3,1)] return(tab) } biomaRt/R/methods-Mart.R0000644000175400017540000001057113175713423016077 0ustar00biocbuildbiocbuildsetMethod("show",signature(object="Mart"), function(object){ dbase <- ifelse(nchar(object@biomart) != 0, yes = paste(" Using the", object@biomart, "BioMart database"), no = " No database selected.") dset <- ifelse(nchar(object@dataset) != 0, yes = paste(" Using the", object@dataset, "dataset"), no = " No dataset selected.") res <- paste("Object of class 'Mart':", dbase, dset, sep="\n") cat(res) }) setGeneric("martBM",def=function(obj,...) standardGeneric("martBM")) setMethod("martBM",signature("Mart"), function(obj) obj@biomart) setGeneric("martBM<-", function(obj, value) standardGeneric("martBM<-")) setReplaceMethod("martBM","Mart",function(obj,value){ obj@biomart <- value obj }) setGeneric("martAttributes",def=function(obj,...)standardGeneric("martAttributes")) setMethod("martAttributes",signature("Mart"),function(obj) obj@attributes) setGeneric("martAttributes<-", function(obj, value) standardGeneric("martAttributes<-")) setReplaceMethod("martAttributes","Mart",function(obj,value){ obj@attributes <- value obj }) setGeneric("martAttribPointers",def=function(obj,...)standardGeneric("martAttribPointers")) setMethod("martAttribPointers",signature("Mart"),function(obj) obj@attributePointer) setGeneric("martAttribPointers<-", function(obj, value) standardGeneric("martAttribPointers<-")) setReplaceMethod("martAttribPointers","Mart",function(obj,value){ obj@attributePointer <- value obj }) setGeneric("martFilters",def=function(obj,...)standardGeneric("martFilters")) setMethod("martFilters",signature("Mart"),function(obj) obj@filters) setGeneric("martFilters<-", function(obj, value) standardGeneric("martFilters<-")) setReplaceMethod("martFilters","Mart",function(obj,value){ obj@filters <- value obj }) setGeneric("martDataset",def=function(obj,...)standardGeneric("martDataset")) setMethod("martDataset",signature("Mart"), function(obj) obj@dataset) setGeneric("martDataset<-", function(obj, value) standardGeneric("martDataset<-")) setReplaceMethod("martDataset","Mart",function(obj,value){ obj@dataset <- value obj }) setGeneric("martHost",def=function(obj,...)standardGeneric("martHost")) setMethod("martHost",signature("Mart"), function(obj) obj@host) setGeneric("martArchive",def=function(obj,...)standardGeneric("martArchive")) setMethod("martArchive",signature("Mart"), function(obj) obj@archive) setGeneric("martVSchema",def=function(obj,...)standardGeneric("martVSchema")) setMethod("martVSchema",signature("Mart"), function(obj) obj@vschema) setGeneric("martVSchema<-", function(obj, value) standardGeneric("martVSchema<-")) setReplaceMethod("martVSchema","Mart",function(obj,value){ obj@vschema <- value obj }) ##################################################################### ## new wrappers to enable keys, columns, select and keytypes .keys <- function(x, keytype){ res <- filterOptions(filter=keytype, mart=x) res <- sub("\\]$","",res) res <- sub("^\\[","",res) unlist(strsplit(res, split=",")) } setMethod("keys", "Mart", function(x, keytype, ...){ AnnotationDbi:::smartKeys(x=x, keytype=keytype, ..., FUN=.keys) } ) setMethod("keytypes", "Mart", function(x) listFilters(mart=x)[["name"]] ) setMethod("columns", "Mart", function(x) listAttributes(mart=x)[["name"]] ) ## Arg checking is similar (but more limited) to what is done for getBM setMethod("select", "Mart", function(x, keys, columns, keytype, ...){ if(missing( columns )) stop("Argument 'columns' must be specified.") if(!is.list(keytype) && keytype != "" && missing( keys )) stop("Argument 'keys' must be specified.") if(length(keytype) > 0 && length(keys) == 0) stop("Keys argument contains no data.") if(!(is.character(keytype)) || length(keytype)!=1){ stop("keytype should be single element character vector.") } getBM(attributes=columns, filters=keytype, values=keys, mart=x) } ) biomaRt/R/utilityFunctions.R0000644000175400017540000001676213214572320017130 0ustar00biocbuildbiocbuild ## sometimes results can be returned by getMB() in a different order to we ## asked for them, which messes up the column names. Here we try to match ## results to known attribute names and rename accordingly. .setResultColNames <- function(result, mart, attributes, bmHeader = FALSE) { ## get all avaialble sttributes and ## filter only for the ones we've actually asked for att <- listAttributes(mart, what = c("name", "description")) att <- att[which(att[,'name'] %in% attributes), ] if(length(which(duplicated(att[,'description']))) > length(which(duplicated(att)))) { warning("Cannot unambiguously match attribute names Ignoring bmHeader argument and using biomart description field") return(result) } resultNames = colnames(result) ## match the returned column names with the attribute names matches <- match(resultNames, att[,2], NA) if(any(is.na(matches))) { warning("Problems assigning column names.", "Currently using the biomart description field.", "You may wish to set these manually.") return(result) } ## if we want to use the attribute names we specified, do this, ## otherwise we use the header returned with the query if(!bmHeader) { colnames(result) = att[matches, 1] } ## now put things in the order we actually asked for the attributes in result <- result[, match(att[matches,1], attributes), drop=FALSE] return(result) } ## BioMart doesn't work well if the list of values provided to a filter is ## longer than 500 values. It returns only a subset of the requested data ## and does so silently! This function is designed to take a list of provided ## filters, and split any longer than 'maxChunkSize'. It operates recursively ## incase there are multiple filters that need splitting, and should ensure ## all possible groupings of filters are retained. .splitValues <- function(valuesList, maxChunkSize = 500) { vLength <- vapply(valuesList[[1]], FUN = length, FUN.VALUE = integer(1)) if(all(vLength <= maxChunkSize)) { return(valuesList) } else { ## pick the next filter to split vIdx <- min(which(vLength > maxChunkSize)) nchunks <- (vLength[vIdx] %/% maxChunkSize) + 1 splitIdx <- rep(1:nchunks, each = ceiling(vLength[vIdx] / nchunks))[ 1:vLength[vIdx] ] ## a new list we will populate with the chunks tmpList <- list() for(i in 1:nchunks) { for( j in 1:length(valuesList) ) { listIdx <- ((i - 1) * length(valuesList)) + j tmpList[[ listIdx ]] <- valuesList[[j]] tmpList[[ listIdx ]][[ vIdx ]] <- tmpList[[ listIdx ]][[ vIdx ]][which(splitIdx == i)] } } ## recursively call the function to process next filter valuesList <- .splitValues(tmpList) } return(valuesList) } ## Creating the filter XML for a single chunk of values. Returns a character ## vector containing the XML lines for all specified filters & their ## attributes spliced together into a single string. .createFilterXMLchunk <- function(filterChunk, mart) { individualFilters <- vapply(names(filterChunk), FUN = function(filter, values, mart) { ## if the filter exists and is boolean we do this if(filter %in% listFilters(mart, what = "name") && grepl('boolean', filterType(filter = filter, mart = mart)) ) { if(!is.logical(values[[filter]])) stop("biomaRt error:\n", filter, " is a boolean filter and needs a corresponding logical value of TRUE or FALSE to indicate if the query should retrieve all data that fulfill the boolean or alternatively that all data that not fulfill the requirement should be retrieved.") val <- ifelse(values[[filter]], yes = 0, no = 1) val <- paste0("' excluded = \"", val, "\" ") } else { ## otherwise the filter isn't boolean, or doesn't exist if(is.numeric(values[[filter]])) values[[filter]] <- as.integer(values[[filter]]) val <- paste0(values[[filter]], collapse = ",") val <- paste0("' value = '", val, "' ") } filterXML <- paste0(" %\VignetteIndexEntry{The biomaRt users guide} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- ```{r setup, cache = F, echo = FALSE} knitr::opts_chunk$set(error = TRUE) ``` # Introduction In recent years a wealth of biological data has become available in public data repositories. Easy access to these valuable data resources and firm integration with data analysis is needed for comprehensive bioinformatics data analysis. The `r Biocpkg("biomaRt")` package, provides an interface to a growing collection of databases implementing the [BioMart software suite](http://www.biomart.org). The package enables retrieval of large amounts of data in a uniform way without the need to know the underlying database schemas or write complex SQL queries. Examples of BioMart databases are Ensembl, Uniprot and HapMap. These major databases give `r Biocpkg("biomaRt")` users direct access to a diverse set of data and enable a wide range of powerful online queries from R. # Selecting a BioMart database and dataset Every analysis with `r Biocpkg("biomaRt")` starts with selecting a BioMart database to use. A first step is to check which BioMart web services are available. The function `listMarts()` will display all available BioMart web services ```{r annotate,echo=FALSE} ## library("annotate") options(width=120) ``` ```{r biomaRt} library("biomaRt") listMarts() ``` Note: if the function `useMart()` runs into proxy problems you should set your proxy first before calling any `r Biocpkg("biomaRt")` functions. You can do this using the Sys.putenv command: ```{r putenv, eval = FALSE} Sys.setenv("http_proxy" = "http://my.proxy.org:9999") ``` Some users have reported that the workaround above does not work, in this case an alternative proxy solution below can be tried: ```{r rCurlOptions, eval = FALSE} options(RCurlOptions = list(proxy="uscache.kcc.com:80",proxyuserpwd="------:-------")) ``` The `useMart()` function can now be used to connect to a specified BioMart database, this must be a valid name given by `listMarts()`. In the next example we choose to query the Ensembl BioMart database. ```{r ensembl1} ensembl=useMart("ensembl") ``` BioMart databases can contain several datasets, for Ensembl every species is a different dataset. In a next step we look at which datasets are available in the selected BioMart by using the function `listDatasets()`. ```{r listDatasets} listDatasets(ensembl) ``` To select a dataset we can update the `Mart` object using the function `useDataset()`. In the example below we choose to use the hsapiens dataset. ```{r ensembl2, eval=TRUE} ensembl = useDataset("hsapiens_gene_ensembl",mart=ensembl) ``` Or alternatively if the dataset one wants to use is known in advance, we can select a BioMart database and dataset in one step by: ```{r ensembl3} ensembl = useMart("ensembl",dataset="hsapiens_gene_ensembl") ``` # How to build a biomaRt query The `getBM()` function has three arguments that need to be introduced: filters, attributes and values. *Filters* define a restriction on the query. For example you want to restrict the output to all genes located on the human X chromosome then the filter *chromosome_name* can be used with value 'X'. The `listFilters()` function shows you all available filters in the selected dataset. ```{r filters} filters = listFilters(ensembl) filters[1:5,] ``` *Attributes* define the values we are interested in to retrieve. For example we want to retrieve the gene symbols or chromosomal coordinates. The `listAttributes()` function displays all available attributes in the selected dataset. ```{r attributes} attributes = listAttributes(ensembl) attributes[1:5,] ``` The `getBM()` function is the main query function in `r Biocpkg("biomaRt")`. It has four main arguments: * `attributes`: is a vector of attributes that one wants to retrieve (= the output of the query). * `filters`: is a vector of filters that one wil use as input to the query. * `values`: a vector of values for the filters. In case multple filters are in use, the values argument requires a list of values where each position in the list corresponds to the position of the filters in the filters argument (see examples below). * `mart`: is an object of class `Mart`, which is created by the `useMart()` function. *Note: for some frequently used queries to Ensembl, wrapper functions are available: `getGene()` and `getSequence()`. These functions call the `getBM()` function with hard coded filter and attribute names.* Now that we selected a BioMart database and dataset, and know about attributes, filters, and the values for filters; we can build a `r Biocpkg("biomaRt")` query. Let's make an easy query for the following problem: We have a list of Affymetrix identifiers from the u133plus2 platform and we want to retrieve the corresponding EntrezGene identifiers using the Ensembl mappings. The u133plus2 platform will be the filter for this query and as values for this filter we use our list of Affymetrix identifiers. As output (attributes) for the query we want to retrieve the EntrezGene and u133plus2 identifiers so we get a mapping of these two identifiers as a result. The exact names that we will have to use to specify the attributes and filters can be retrieved with the `listAttributes()` and `listFilters()` function respectively. Let's now run the query: ```{r getBM1, echo=TRUE,eval=TRUE} affyids=c("202763_at","209310_s_at","207500_at") getBM(attributes=c('affy_hg_u133_plus_2', 'entrezgene'), filters = 'affy_hg_u133_plus_2', values = affyids, mart = ensembl) ``` # Examples of biomaRt queries In the sections below a variety of example queries are described. Every example is written as a task, and we have to come up with a `r Biocpkg("biomaRt")` solution to the problem. ## Annotate a set of Affymetrix identifiers with HUGO symbol and chromosomal locations of corresponding genes We have a list of Affymetrix hgu133plus2 identifiers and we would like to retrieve the HUGO gene symbols, chromosome names, start and end positions and the bands of the corresponding genes. The `listAttributes()` and the `listFilters()` functions give us an overview of the available attributes and filters and we look in those lists to find the corresponding attribute and filter names we need. For this query we'll need the following attributes: hgnc_symbol, chromsome_name, start_position, end_position, band and affy_hg_u133_plus_2 (as we want these in the output to provide a mapping with our original Affymetrix input identifiers. There is one filter in this query which is the affy_hg_u133_plus_2 filter as we use a list of Affymetrix identifiers as input. Putting this all together in the `getBM()` and performing the query gives: ```{r task1, echo=TRUE,eval=TRUE} affyids=c("202763_at","209310_s_at","207500_at") getBM(attributes = c('affy_hg_u133_plus_2', 'hgnc_symbol', 'chromosome_name', 'start_position', 'end_position', 'band'), filters = 'affy_hg_u133_plus_2', values = affyids, mart = ensembl) ``` ## Annotate a set of EntrezGene identifiers with GO annotation In this task we start out with a list of EntrezGene identiers and we want to retrieve GO identifiers related to biological processes that are associated with these entrezgene identifiers. Again we look at the output of `listAttributes()` and `listFilters()` to find the filter and attributes we need. Then we construct the following query: ```{r task2, echo=TRUE,eval=TRUE} entrez=c("673","837") goids = getBM(attributes = c('entrezgene', 'go_id'), filters = 'entrezgene', values = entrez, mart = ensembl) head(goids) ``` ## Retrieve all HUGO gene symbols of genes that are located on chromosomes 17,20 or Y, and are associated with specific GO terms The GO terms we are interested in are: **GO:0051330**, **GO:0000080**, **GO:0000114**, **GO:0000082**. The key to performing this query is to understand that the `getBM()` function enables you to use more than one filter at the same time. In order to do this, the filter argument should be a vector with the filter names. The values should be a list, where the first element of the list corresponds to the first filter and the second list element to the second filter and so on. The elements of this list are vectors containing the possible values for the corresponding filters. ```{r task3, echo=TRUE,eval=TRUE} go=c("GO:0051330","GO:0000080","GO:0000114","GO:0000082") chrom=c(17,20,"Y") getBM(attributes= "hgnc_symbol", filters=c("go_id","chromosome_name"), values=list(go, chrom), mart=ensembl) ``` ## Annotate set of idenfiers with INTERPRO protein domain identifiers In this example we want to annotate the following two RefSeq identifiers: **NM_005359** and **NM_000546** with INTERPRO protein domain identifiers and a description of the protein domains. ```{r task4, echo=TRUE,eval=TRUE} refseqids = c("NM_005359","NM_000546") ipro = getBM(attributes=c("refseq_mrna","interpro","interpro_description"), filters="refseq_mrna", values=refseqids, mart=ensembl) ipro ``` ## Select all Affymetrix identifiers on the hgu133plus2 chip and Ensembl gene identifiers for genes located on chromosome 16 between basepair 1100000 and 1250000. In this example we will again use multiple filters: *chromosome_name*, *start*, and *end* as we filter on these three conditions. Note that when a chromosome name, a start position and an end position are jointly used as filters, the BioMart webservice interprets this as return everything from the given chromosome between the given start and end positions. ```{r task5, eval = TRUE} getBM(attributes = c('affy_hg_u133_plus_2','ensembl_gene_id'), filters = c('chromosome_name','start','end'), values = list(16,1100000,1250000), mart = ensembl) ``` ## Retrieve all entrezgene identifiers and HUGO gene symbols of genes which have a "MAP kinase activity" GO term associated with it. The GO identifier for MAP kinase activity is **GO:0004707**. In our query we will use *go_id* as our filter, and *entrezgene* and *hgnc_symbol* as attributes. Here's the query: ```{r task6, echo=TRUE, eval = TRUE} getBM(attributes = c('entrezgene','hgnc_symbol'), filters = 'go', values = 'GO:0004707', mart = ensembl) ``` ## Given a set of EntrezGene identifiers, retrieve 100bp upstream promoter sequences All sequence related queries to Ensembl are available through the `getSequence()` wrapper function. `getBM()` can also be used directly to retrieve sequences but this can get complicated so using getSequence is recommended. Sequences can be retrieved using the `getSequence()` function either starting from chromosomal coordinates or identifiers. The chromosome name can be specified using the *chromosome* argument. The *start* and *end* arguments are used to specify *start* and *end* positions on the chromosome. The type of sequence returned can be specified by the *seqType* argument which takes the following values: * *cdna* * *peptide* for protein sequences * *3utr* for 3' UTR sequences * *5utr* for 5' UTR sequences * *gene_exon* for exon sequences only * *transcript_exon* for transcript specific exonic sequences only * *transcript_exon_intron* gives the full unspliced transcript, that is exons + introns * *gene_exon_intron* gives the exons + introns of a gene * *coding* gives the coding sequence only * *coding_transcript_flank* gives the flanking region of the transcript including the UTRs, this must be accompanied with a given value for the upstream or downstream attribute * *coding_gene_flank* gives the flanking region of the gene including the UTRs, this must be accompanied with a given value for the upstream or downstream attribute * *transcript_flank* gives the flanking region of the transcript exculding the UTRs, this must be accompanied with a given value for the upstream or downstream attribute * *gene_flank* gives the flanking region of the gene excluding the UTRs, this must be accompanied with a given value for the upstream or downstream attribute In MySQL mode the `getSequence()` function is more limited and the sequence that is returned is the 5' to 3'+ strand of the genomic sequence, given a chromosome, as start and an end position. This task requires us to retrieve 100bp upstream promoter sequences from a set of EntrzGene identifiers. The *type* argument in `getSequence()` can be thought of as the filter in this query and uses the same input names given by `listFilters()`. In our query we use entrezgene for the type argument. Next we have to specify which type of sequences we want to retrieve, here we are interested in the sequences of the promoter region, starting right next to the coding start of the gene. Setting the *seqType* to coding_gene_flank will give us what we need. The *upstream* argument is used to specify how many bp of upstream sequence we want to retrieve, here we'll retrieve a rather short sequence of 100bp. Putting this all together in `getSequence()` gives: ```{r task7, eval=TRUE} entrez=c("673","7157","837") getSequence(id = entrez, type="entrezgene", seqType="coding_gene_flank", upstream=100, mart=ensembl) ``` ## Retrieve all 5' UTR sequences of all genes that are located on chromosome 3 between the positions 185,514,033 and 185,535,839 As described in the provious task getSequence can also use chromosomal coordinates to retrieve sequences of all genes that lie in the given region. We also have to specify which type of identifier we want to retrieve together with the sequences, here we choose for entrezgene identifiers. ```{r task8, echo=TRUE,eval=TRUE} utr5 = getSequence(chromosome=3, start=185514033, end=185535839, type="entrezgene", seqType="5utr", mart=ensembl) utr5 ``` ## Retrieve protein sequences for a given list of EntrezGene identifiers In this task the type argument specifies which type of identifiers we are using. To get an overview of other valid identifier types we refer to the `listFilters()` function. ```{r task9, echo=TRUE, eval=TRUE} protein = getSequence(id=c(100, 5728), type="entrezgene", seqType="peptide", mart=ensembl) protein ``` ## Retrieve known SNPs located on the human chromosome 8 between positions 148350 and 148612 For this example we'll first have to connect to a different BioMart database, namely snp. ```{r task10, echo=TRUE, eval=TRUE} snpmart = useMart(biomart = "ENSEMBL_MART_SNP", dataset="hsapiens_snp") ``` The `listAttributes()` and `listFilters()` functions give us an overview of the available attributes and filters. From these we need: *refsnp_id*, *allele*, *chrom_start* and *chrom_strand* as attributes; and as filters we'll use: *chrom_start*, *chrom_end* and *chr_name*. Note that when a chromosome name, a start position and an end position are jointly used as filters, the BioMart webservice interprets this as return everything from the given chromosome between the given start and end positions. Putting our selected attributes and filters into getBM gives: ```{r task10b} getBM(attributes = c('refsnp_id','allele','chrom_start','chrom_strand'), filters = c('chr_name','start','end'), values = list(8,148350,148612), mart = snpmart) ``` ## Given the human gene TP53, retrieve the human chromosomal location of this gene and also retrieve the chromosomal location and RefSeq id of its homolog in mouse. The `getLDS()` (Get Linked Dataset) function provides functionality to link 2 BioMart datasets which each other and construct a query over the two datasets. In Ensembl, linking two datasets translates to retrieving homology data across species. The usage of getLDS is very similar to `getBM()`. The linked dataset is provided by a separate `Mart` object and one has to specify filters and attributes for the linked dataset. Filters can either be applied to both datasets or to one of the datasets. Use the listFilters and listAttributes functions on both `Mart` objects to find the filters and attributes for each dataset (species in Ensembl). The attributes and filters of the linked dataset can be specified with the attributesL and filtersL arguments. Entering all this information into `getLDS()` gives: ```{r getLDS} human = useMart("ensembl", dataset = "hsapiens_gene_ensembl") mouse = useMart("ensembl", dataset = "mmusculus_gene_ensembl") getLDS(attributes = c("hgnc_symbol","chromosome_name", "start_position"), filters = "hgnc_symbol", values = "TP53",mart = human, attributesL = c("refseq_mrna","chromosome_name","start_position"), martL = mouse) ``` # Using archived versions of Ensembl It is possible to query archived versions of Ensembl through `r Biocpkg("biomaRt")`. `r Biocpkg("biomaRt")` provides the function `listEnsemblArchives()` to view the available archives. This function takes no arguments, and produces a table containing the names of the available archived versions, the date they were first available, and the URL where they can be accessed. ```{r archiveMarts, echo = TRUE, eval = TRUE} listEnsemblArchives() ``` Alternatively, one can use the website to find archived version. From the main page scroll down the bottom of the page, click on 'view in Archive' and select the archive you need. *You will notice that there is an archive URL even for the current release of Ensembl. It can be useful to use this if you wish to ensure that script you write now will return exactly the same results in the future. Using `www.ensembl.org` will always access the current release, and so the data retrieved may change over time as new releases come out.* Whichever method you use to find the URL of the archive you wish to query, copy the url and use that in the `host` argument as shown below to connect to the specified BioMart database. The example below shows how to query Ensembl 54. ```{r archiveMarts3, echo = TRUE, eval = TRUE} listMarts(host = 'may2009.archive.ensembl.org') ensembl54 <- useMart(host='may2009.archive.ensembl.org', biomart='ENSEMBL_MART_ENSEMBL', dataset='hsapiens_gene_ensembl') ``` # Using a BioMart other than Ensembl To demonstrate the use of the `r Biocpkg("biomaRt")` package with non-Ensembl databases the next query is performed using the Wormbase ParaSite BioMart. In this example, we use the `listMarts()` function to find the name of the available marts, given the URL of Wormbase. We use this to connect to Wormbase BioMart, find and select the gene dataset, and print the first 6 available attributes and filters. Then we use a list of gene names as filter and retrieve associated transcript IDs and the transcript biotype. ```{r wormbase, echo=TRUE, eval=TRUE} listMarts(host = "parasite.wormbase.org") wormbase = useMart(biomart = "parasite_mart", host = "parasite.wormbase.org") listDatasets(wormbase) wormbase <- useDataset(mart = wormbase, dataset = "wbps_gene") head(listFilters(wormbase)) head(listAttributes(wormbase)) getBM(attributes = c("external_gene_id", "wbps_transcript_id", "transcript_biotype"), filters="gene_name", values=c("unc-26","his-33"), mart=wormbase) ``` # biomaRt helper functions This section describes a set of `r Biocpkg("biomaRt")` helper functions that can be used to export FASTA format sequences, retrieve values for certain filters and exploring the available filters and attributes in a more systematic manner. ## exportFASTA The data.frames obtained by the getSequence function can be exported to FASTA files using the `exportFASTA()` function. One has to specify the data.frame to export and the filename using the file argument. ## Finding out more information on filters ### filterType Boolean filters need a value TRUE or FALSE in `r Biocpkg("biomaRt")`. Setting the value TRUE will include all information that fulfill the filter requirement. Setting FALSE will exclude the information that fulfills the filter requirement and will return all values that don't fulfill the filter. For most of the filters, their name indicates if the type is a boolean or not and they will usually start with "with". However this is not a rule and to make sure you got the type right you can use the function `filterType()` to investigate the type of the filter you want to use. ```{r filterType} filterType("with_affy_hg_u133_plus_2",ensembl) ``` ### filterOptions Some filters have a limited set of values that can be given to them. To know which values these are one can use the `filterOptions()` function to retrieve the predetermed values of the respective filter. ```{r filterOptions} filterOptions("biotype",ensembl) ``` If there are no predetermed values e.g. for the entrezgene filter, then `filterOptions()` will return the type of filter it is. And most of the times the filter name or it's description will suggest what values one case use for the respective filter (e.g. entrezgene filter will work with enterzgene identifiers as values) ## Attribute Pages For large BioMart databases such as Ensembl, the number of attributes displayed by the `listAttributes()` function can be very large. In BioMart databases, attributes are put together in pages, such as sequences, features, homologs for Ensembl. An overview of the attributes pages present in the respective BioMart dataset can be obtained with the `attributePages()` function. ```{r attributePages} pages = attributePages(ensembl) pages ``` To show us a smaller list of attributes which belong to a specific page, we can now specify this in the `listAttributes()` function. *The set of attributes is still quite long, so we use `head()` to show only the first few items here.* ```{r listAttributes} head(listAttributes(ensembl, page="feature_page")) ``` We now get a short list of attributes related to the region where the genes are located. # Local BioMart databases The `r Biocpkg("biomaRt")` package can be used with a local install of a public BioMart database or a locally developed BioMart database and web service. In order for `r Biocpkg("biomaRt")` to recognize the database as a BioMart, make sure that the local database you create has a name conform with ` database_mart_version ` where database is the name of the database and version is a version number. No more underscores than the ones showed should be present in this name. A possible name is for example ` ensemblLocal_mart_46 `. ## Minimum requirements for local database installation More information on installing a local copy of a BioMart database or develop your own BioMart database and webservice can be found on Once the local database is installed you can use `r Biocpkg("biomaRt")` on this database by: ```{r localCopy, eval = FALSE} listMarts(host="www.myLocalHost.org", path="/myPathToWebservice/martservice") mart=useMart("nameOfMyMart",dataset="nameOfMyDataset",host="www.myLocalHost.org", path="/myPathToWebservice/martservice") ``` For more information on how to install a public BioMart database see: http://www.biomart.org/install.html and follow link databases. # Using `select()` In order to provide a more consistent interface to all annotations in Bioconductor the `select()`, `columns()`, `keytypes()` and `keys()` have been implemented to wrap some of the existing functionality above. These methods can be called in the same manner that they are used in other parts of the project except that instead of taking a `AnnotationDb` derived class they take instead a `Mart` derived class as their 1st argument. Otherwise usage should be essentially the same. You still use `columns()` to discover things that can be extracted from a `Mart`, and `keytypes()` to discover which things can be used as keys with `select()`. ```{r columnsAndKeyTypes} mart <- useMart(dataset="hsapiens_gene_ensembl",biomart='ensembl') head(keytypes(mart), n=3) head(columns(mart), n=3) ``` And you still can use `keys()` to extract potential keys, for a particular key type. ```{r keys1} k = keys(mart, keytype="chromosome_name") head(k, n=3) ``` When using `keys()`, you can even take advantage of the extra arguments that are available for others keys methods. ```{r keys2} k = keys(mart, keytype="chromosome_name", pattern="LRG") head(k, n=3) ``` Unfortunately the `keys()` method will not work with all key types because they are not all supported. But you can still use `select()` here to extract columns of data that match a particular set of keys (this is basically a wrapper for `getBM()`). ```{r select} affy=c("202763_at","209310_s_at","207500_at") select(mart, keys=affy, columns=c('affy_hg_u133_plus_2','entrezgene'), keytype='affy_hg_u133_plus_2') ``` So why would we want to do this when we already have functions like `getBM()`? For two reasons: 1) for people who are familiar with select and it's helper methods, they can now proceed to use `r Biocpkg("biomaRt")` making the same kinds of calls that are already familiar to them and 2) because the select method is implemented in many places elsewhere, the fact that these methods are shared allows for more convenient programmatic access of all these resources. An example of a package that takes advantage of this is the `r Biocpkg("OrganismDbi")` package. Where several packages can be accessed as if they were one resource. # Session Info ```{r sessionInfo} sessionInfo() warnings() ``` biomaRt/inst/doc/biomaRt.html0000644000175400017540000307305213214603700017210 0ustar00biocbuildbiocbuild The biomaRt users guide

    Contents

    1 Introduction

    In recent years a wealth of biological data has become available in public data repositories. Easy access to these valuable data resources and firm integration with data analysis is needed for comprehensive bioinformatics data analysis. The biomaRt package, provides an interface to a growing collection of databases implementing the BioMart software suite. The package enables retrieval of large amounts of data in a uniform way without the need to know the underlying database schemas or write complex SQL queries. Examples of BioMart databases are Ensembl, Uniprot and HapMap. These major databases give biomaRt users direct access to a diverse set of data and enable a wide range of powerful online queries from R.

    2 Selecting a BioMart database and dataset

    Every analysis with biomaRt starts with selecting a BioMart database to use. A first step is to check which BioMart web services are available. The function listMarts() will display all available BioMart web services

    library("biomaRt")
    listMarts()
    ##                biomart               version
    ## 1 ENSEMBL_MART_ENSEMBL      Ensembl Genes 91
    ## 2   ENSEMBL_MART_MOUSE      Mouse strains 91
    ## 3     ENSEMBL_MART_SNP  Ensembl Variation 91
    ## 4 ENSEMBL_MART_FUNCGEN Ensembl Regulation 91

    Note: if the function useMart() runs into proxy problems you should set your proxy first before calling any biomaRt functions.
    You can do this using the Sys.putenv command:

    Sys.setenv("http_proxy" = "http://my.proxy.org:9999")

    Some users have reported that the workaround above does not work, in this case an alternative proxy solution below can be tried:

    options(RCurlOptions = list(proxy="uscache.kcc.com:80",proxyuserpwd="------:-------"))

    The useMart() function can now be used to connect to a specified BioMart database, this must be a valid name given by listMarts(). In the next example we choose to query the Ensembl BioMart database.

    ensembl=useMart("ensembl")

    BioMart databases can contain several datasets, for Ensembl every species is a different dataset. In a next step we look at which datasets are available in the selected BioMart by using the function listDatasets().

    listDatasets(ensembl)
    ##                           dataset                                                  description                 version
    ## 1       amelanoleuca_gene_ensembl                                        Panda genes (ailMel1)                 ailMel1
    ## 2             dordii_gene_ensembl                                Kangaroo rat genes (Dord_2.0)                Dord_2.0
    ## 3            mpahari_gene_ensembl                          Shrew mouse genes (PAHARI_EIJ_v1.1)         PAHARI_EIJ_v1.1
    ## 4          trubripes_gene_ensembl                                        Fugu genes (FUGU 4.0)                FUGU 4.0
    ## 5           pmarinus_gene_ensembl                                 Lamprey genes (Pmarinus_7.0)            Pmarinus_7.0
    ## 6          sharrisii_gene_ensembl                       Tasmanian devil genes (Devil_ref v7.0)          Devil_ref v7.0
    ## 7        nleucogenys_gene_ensembl                                      Gibbon genes (Nleu_3.0)                Nleu_3.0
    ## 8            ggallus_gene_ensembl                            Chicken genes (Gallus_gallus-5.0)       Gallus_gallus-5.0
    ## 9           olatipes_gene_ensembl                                          Medaka genes (HdrR)                    HdrR
    ## 10        pcoquereli_gene_ensembl                           Coquerel's sifaka genes (Pcoq_1.0)                Pcoq_1.0
    ## 11         pvampyrus_gene_ensembl                                      Megabat genes (pteVam1)                 pteVam1
    ## 12     acarolinensis_gene_ensembl                               Anole lizard genes (AnoCar2.0)               AnoCar2.0
    ## 13           hfemale_gene_ensembl              Naked mole-rat female genes (HetGla_female_1.0)       HetGla_female_1.0
    ## 14        ogarnettii_gene_ensembl                                     Bushbaby genes (OtoGar3)                 OtoGar3
    ## 15     tnigroviridis_gene_ensembl                              Tetraodon genes (TETRAODON 8.0)           TETRAODON 8.0
    ## 16       falbicollis_gene_ensembl                                Flycatcher genes (FicAlb_1.4)              FicAlb_1.4
    ## 17     mfascicularis_gene_ensembl          Crab-eating macaque genes (Macaca_fascicularis_5.0) Macaca_fascicularis_5.0
    ## 18          neugenii_gene_ensembl                                     Wallaby genes (Meug_1.0)                Meug_1.0
    ## 19           ngalili_gene_ensembl Upper Galilee mountains blind mole rat genes (S.galili_v1.0)           S.galili_v1.0
    ## 20          saraneus_gene_ensembl                                        Shrew genes (sorAra1)                 sorAra1
    ## 21        ttruncatus_gene_ensembl                                      Dolphin genes (turTru1)                 turTru1
    ## 22        cpalliatus_gene_ensembl                           Angola colobus genes (Cang.pa_1.0)             Cang.pa_1.0
    ## 23       scerevisiae_gene_ensembl                     Saccharomyces cerevisiae genes (R64-1-1)                 R64-1-1
    ## 24        mgallopavo_gene_ensembl                                   Turkey genes (Turkey_2.01)             Turkey_2.01
    ## 25     cintestinalis_gene_ensembl                                    C.intestinalis genes (KH)                      KH
    ## 26 itridecemlineatus_gene_ensembl                                   Squirrel genes (SpeTri2.0)               SpeTri2.0
    ## 27            odegus_gene_ensembl                                       Degu genes (OctDeg1.0)               OctDeg1.0
    ## 28         psinensis_gene_ensembl                  Chinese softshell turtle genes (PelSin_1.0)              PelSin_1.0
    ## 29       fdamarensis_gene_ensembl                             Damara mole rat genes (DMR_v1.0)                DMR_v1.0
    ## 30           gmorhua_gene_ensembl                                          Cod genes (gadMor1)                 gadMor1
    ## 31          pformosa_gene_ensembl                  Amazon molly genes (Poecilia_formosa-5.1.2)  Poecilia_formosa-5.1.2
    ## 32         csavignyi_gene_ensembl                                  C.savignyi genes (CSAV 2.0)                CSAV 2.0
    ## 33         ppaniscus_gene_ensembl                                     Bonobo genes (panpan1.1)               panpan1.1
    ## 34          mauratus_gene_ensembl                             Golden Hamster genes (MesAur1.0)               MesAur1.0
    ## 35        choffmanni_gene_ensembl                                        Sloth genes (choHof1)                 choHof1
    ## 36          csabaeus_gene_ensembl                                 Vervet-AGM genes (ChlSab1.1)               ChlSab1.1
    ## 37        lchalumnae_gene_ensembl                                   Coelacanth genes (LatCha1)                 LatCha1
    ## 38       rnorvegicus_gene_ensembl                                         Rat genes (Rnor_6.0)                Rnor_6.0
    ## 39          cjacchus_gene_ensembl                              Marmoset genes (C_jacchus3.2.1)          C_jacchus3.2.1
    ## 40          pbairdii_gene_ensembl                Northern American deer mouse genes (Pman_1.0)                Pman_1.0
    ## 41           mcaroli_gene_ensembl                         Ryukyu mouse genes (CAROLI_EIJ_v1.1)         CAROLI_EIJ_v1.1
    ## 42          hsapiens_gene_ensembl                                     Human genes (GRCh38.p10)              GRCh38.p10
    ## 43           btaurus_gene_ensembl                                           Cow genes (UMD3.1)                  UMD3.1
    ## 44            drerio_gene_ensembl                                     Zebrafish genes (GRCz10)                  GRCz10
    ## 45             mfuro_gene_ensembl                                  Ferret genes (MusPutFur1.0)            MusPutFur1.0
    ## 46           caperea_gene_ensembl                        Brazilian guinea pig genes (CavAp1.0)                CavAp1.0
    ## 47             catys_gene_ensembl                              Sooty mangabey genes (Caty_1.0)                Caty_1.0
    ## 48       mnemestrina_gene_ensembl                          Pig-tailed macaque genes (Mnem_1.0)                Mnem_1.0
    ## 49          ggorilla_gene_ensembl                                      Gorilla genes (gorGor4)                 gorGor4
    ## 50         csyrichta_gene_ensembl                       Tarsier genes (Tarsius_syrichta-2.0.1)  Tarsius_syrichta-2.0.1
    ## 51            vpacos_gene_ensembl                                       Alpaca genes (vicPac1)                 vicPac1
    ## 52      ptroglodytes_gene_ensembl                               Chimpanzee genes (Pan_tro_3.0)             Pan_tro_3.0
    ## 53            rbieti_gene_ensembl                  Black snub-nosed monkey genes (ASM169854v1)             ASM169854v1
    ## 54        tbelangeri_gene_ensembl                                   Tree Shrew genes (tupBel1)                 tupBel1
    ## 55        mdomestica_gene_ensembl                                      Opossum genes (monDom5)                 monDom5
    ## 56         oprinceps_gene_ensembl                                   Pika genes (OchPri2.0-Ens)           OchPri2.0-Ens
    ## 57      mleucophaeus_gene_ensembl                                    Drill genes (Mleu.le_1.0)             Mleu.le_1.0
    ## 58        ccapucinus_gene_ensembl                          Capuchin genes (Cebus_imitator-1.0)      Cebus_imitator-1.0
    ## 59          mmulatta_gene_ensembl                                   Macaque genes (Mmul_8.0.1)              Mmul_8.0.1
    ## 60        mlucifugus_gene_ensembl                                   Microbat genes (Myoluc2.0)               Myoluc2.0
    ## 61        gaculeatus_gene_ensembl                                 Stickleback genes (BROAD S1)                BROAD S1
    ## 62         lafricana_gene_ensembl                                   Elephant genes (Loxafr3.0)               Loxafr3.0
    ## 63         etelfairi_gene_ensembl                        Lesser hedgehog tenrec genes (TENREC)                  TENREC
    ## 64         loculatus_gene_ensembl                                  Spotted gar genes (LepOcu1)                 LepOcu1
    ## 65        xmaculatus_gene_ensembl                                Platyfish genes (Xipmac4.4.2)             Xipmac4.4.2
    ## 66          celegans_gene_ensembl                      Caenorhabditis elegans genes (WBcel235)                WBcel235
    ## 67           panubis_gene_ensembl                                Olive baboon genes (Panu_3.0)                Panu_3.0
    ## 68             hmale_gene_ensembl                       Naked mole-rat male genes (HetGla_1.0)              HetGla_1.0
    ## 69            oaries_gene_ensembl                                       Sheep genes (Oar_v3.1)                Oar_v3.1
    ## 70          mmurinus_gene_ensembl                                 Mouse Lemur genes (Mmur_3.0)                Mmur_3.0
    ## 71         pcapensis_gene_ensembl                                        Hyrax genes (proCap1)                 proCap1
    ## 72        ocuniculus_gene_ensembl                                     Rabbit genes (OryCun2.0)               OryCun2.0
    ## 73    aplatyrhynchos_gene_ensembl                                    Duck genes (BGI_duck_1.0)            BGI_duck_1.0
    ## 74        anancymaae_gene_ensembl                           Ma's night monkey genes (Anan_2.0)                Anan_2.0
    ## 75           pabelii_gene_ensembl                                      Orangutan genes (PPYG2)                   PPYG2
    ## 76         mspreteij_gene_ensembl                         Mouse SPRET/EiJ genes (SPRET_EiJ_v1)            SPRET_EiJ_v1
    ## 77        rroxellana_gene_ensembl                     Golden snub-nosed monkey genes (Rrox_v1)                 Rrox_v1
    ## 78          jjaculus_gene_ensembl                     Lesser Egyptian jerboa genes (JacJac1.0)               JacJac1.0
    ## 79        cporcellus_gene_ensembl                                 Guinea Pig genes (Cavpor3.0)               Cavpor3.0
    ## 80      mochrogaster_gene_ensembl                               Prairie vole genes (MicOch1.0)               MicOch1.0
    ## 81     dmelanogaster_gene_ensembl                                       Fruitfly genes (BDGP6)                   BDGP6
    ## 82         clanigera_gene_ensembl                     Long-tailed chinchilla genes (ChiLan1.0)               ChiLan1.0
    ## 83       cfamiliaris_gene_ensembl                                        Dog genes (CanFam3.1)               CanFam3.1
    ## 84        cchok1gshd_gene_ensembl                 Chinese hamster CHOK1GS genes (CHOK1GS_HDv1)            CHOK1GS_HDv1
    ## 85       xtropicalis_gene_ensembl                                      Xenopus genes (JGI 4.2)                 JGI 4.2
    ## 86        oniloticus_gene_ensembl                                    Tilapia genes (Orenil1.0)               Orenil1.0
    ## 87     dnovemcinctus_gene_ensembl                                  Armadillo genes (Dasnov3.0)               Dasnov3.0
    ## 88           sscrofa_gene_ensembl                                      Pig genes (Sscrofa11.1)             Sscrofa11.1
    ## 89         ecaballus_gene_ensembl                                      Horse genes (Equ Cab 2)               Equ Cab 2
    ## 90         mmusculus_gene_ensembl                                      Mouse genes (GRCm38.p5)               GRCm38.p5
    ## 91         oanatinus_gene_ensembl                                       Platypus genes (OANA5)                   OANA5
    ## 92          tguttata_gene_ensembl                              Zebra Finch genes (taeGut3.2.4)             taeGut3.2.4
    ## 93            fcatus_gene_ensembl                                  Cat genes (Felis_catus_8.0)         Felis_catus_8.0
    ## 94        amexicanus_gene_ensembl                                  Cave fish genes (AstMex102)               AstMex102
    ## 95        eeuropaeus_gene_ensembl                                     Hedgehog genes (eriEur1)                 eriEur1
    ## 96           ccrigri_gene_ensembl                    Chinese hamster CriGri genes (CriGri_1.0)              CriGri_1.0
    ## 97      sboliviensis_gene_ensembl                   Bolivian squirrel monkey genes (SaiBol1.0)               SaiBol1.0

    To select a dataset we can update the Mart object using the function useDataset(). In the example below we choose to use the hsapiens dataset.

    ensembl = useDataset("hsapiens_gene_ensembl",mart=ensembl)

    Or alternatively if the dataset one wants to use is known in advance, we can select a BioMart database and dataset in one step by:

    ensembl = useMart("ensembl",dataset="hsapiens_gene_ensembl")

    3 How to build a biomaRt query

    The getBM() function has three arguments that need to be introduced: filters, attributes and values. Filters define a restriction on the query. For example you want to restrict the output to all genes located on the human X chromosome then the filter chromosome_name can be used with value ‘X’. The listFilters() function shows you all available filters in the selected dataset.

    filters = listFilters(ensembl)
    filters[1:5,]
    ##              name              description
    ## 1 chromosome_name Chromosome/scaffold name
    ## 2           start                    Start
    ## 3             end                      End
    ## 4      band_start               Band Start
    ## 5        band_end                 Band End

    Attributes define the values we are interested in to retrieve. For example we want to retrieve the gene symbols or chromosomal coordinates. The listAttributes() function displays all available attributes in the selected dataset.

    attributes = listAttributes(ensembl)
    attributes[1:5,]
    ##                            name                  description         page
    ## 1               ensembl_gene_id               Gene stable ID feature_page
    ## 2       ensembl_gene_id_version       Gene stable ID version feature_page
    ## 3         ensembl_transcript_id         Transcript stable ID feature_page
    ## 4 ensembl_transcript_id_version Transcript stable ID version feature_page
    ## 5            ensembl_peptide_id            Protein stable ID feature_page

    The getBM() function is the main query function in biomaRt. It has four main arguments:

    • attributes: is a vector of attributes that one wants to retrieve (= the output of the query).
    • filters: is a vector of filters that one wil use as input to the query.
    • values: a vector of values for the filters. In case multple filters are in use, the values argument requires a list of values where each position in the list corresponds to the position of the filters in the filters argument (see examples below).
    • mart: is an object of class Mart, which is created by the useMart() function.

    Note: for some frequently used queries to Ensembl, wrapper functions are available: getGene() and getSequence(). These functions call the getBM() function with hard coded filter and attribute names.

    Now that we selected a BioMart database and dataset, and know about attributes, filters, and the values for filters; we can build a biomaRt query. Let’s make an easy query for the following problem: We have a list of Affymetrix identifiers from the u133plus2 platform and we want to retrieve the corresponding EntrezGene identifiers using the Ensembl mappings.

    The u133plus2 platform will be the filter for this query and as values for this filter we use our list of Affymetrix identifiers. As output (attributes) for the query we want to retrieve the EntrezGene and u133plus2 identifiers so we get a mapping of these two identifiers as a result. The exact names that we will have to use to specify the attributes and filters can be retrieved with the listAttributes() and listFilters() function respectively. Let’s now run the query:

    affyids=c("202763_at","209310_s_at","207500_at")
    getBM(attributes=c('affy_hg_u133_plus_2', 'entrezgene'), 
          filters = 'affy_hg_u133_plus_2', 
          values = affyids, 
          mart = ensembl)
    ##   affy_hg_u133_plus_2 entrezgene
    ## 1           202763_at        836
    ## 2         209310_s_at        837
    ## 3           207500_at        838

    4 Examples of biomaRt queries

    In the sections below a variety of example queries are described. Every example is written as a task, and we have to come up with a biomaRt solution to the problem.

    4.1 Annotate a set of Affymetrix identifiers with HUGO symbol and chromosomal locations of corresponding genes

    We have a list of Affymetrix hgu133plus2 identifiers and we would like to retrieve the HUGO gene symbols, chromosome names, start and end positions and the bands of the corresponding genes. The listAttributes() and the listFilters() functions give us an overview of the available attributes and filters and we look in those lists to find the corresponding attribute and filter names we need. For this query we’ll need the following attributes: hgnc_symbol, chromsome_name, start_position, end_position, band and affy_hg_u133_plus_2 (as we want these in the output to provide a mapping with our original Affymetrix input identifiers. There is one filter in this query which is the affy_hg_u133_plus_2 filter as we use a list of Affymetrix identifiers as input. Putting this all together in the getBM() and performing the query gives:

    affyids=c("202763_at","209310_s_at","207500_at")
    getBM(attributes = c('affy_hg_u133_plus_2', 'hgnc_symbol', 'chromosome_name',
                       'start_position', 'end_position', 'band'),
          filters = 'affy_hg_u133_plus_2', 
          values = affyids, 
          mart = ensembl)
    ##   affy_hg_u133_plus_2 hgnc_symbol chromosome_name start_position end_position  band
    ## 1           202763_at       CASP3               4      184627696    184649509 q35.1
    ## 2         209310_s_at       CASP4              11      104942866    104969436 q22.3
    ## 3           207500_at       CASP5              11      104994235    105023168 q22.3

    4.2 Annotate a set of EntrezGene identifiers with GO annotation

    In this task we start out with a list of EntrezGene identiers and we want to retrieve GO identifiers related to biological processes that are associated with these entrezgene identifiers. Again we look at the output of listAttributes() and listFilters() to find the filter and attributes we need. Then we construct the following query:

    entrez=c("673","837")
    goids = getBM(attributes = c('entrezgene', 'go_id'), 
                  filters = 'entrezgene', 
                  values = entrez, 
                  mart = ensembl)
    head(goids)
    ##   entrezgene      go_id
    ## 1        673 GO:0000166
    ## 2        673 GO:0004672
    ## 3        673 GO:0004674
    ## 4        673 GO:0005524
    ## 5        673 GO:0006468
    ## 6        673 GO:0010628

    4.3 Retrieve all HUGO gene symbols of genes that are located on chromosomes 17,20 or Y, and are associated with specific GO terms

    The GO terms we are interested in are: GO:0051330, GO:0000080, GO:0000114, GO:0000082. The key to performing this query is to understand that the getBM() function enables you to use more than one filter at the same time. In order to do this, the filter argument should be a vector with the filter names. The values should be a list, where the first element of the list corresponds to the first filter and the second list element to the second filter and so on. The elements of this list are vectors containing the possible values for the corresponding filters.

     go=c("GO:0051330","GO:0000080","GO:0000114","GO:0000082")
     chrom=c(17,20,"Y")
     getBM(attributes= "hgnc_symbol",
            filters=c("go_id","chromosome_name"),
            values=list(go, chrom), mart=ensembl)
    ## Error in getBM(attributes = "hgnc_symbol", filters = c("go_id", "chromosome_name"), : Invalid filters(s): go_id 
    ## Please use the function 'listFilters' to get valid filter names

    4.4 Annotate set of idenfiers with INTERPRO protein domain identifiers

    In this example we want to annotate the following two RefSeq identifiers: NM_005359 and NM_000546 with INTERPRO protein domain identifiers and a description of the protein domains.

    refseqids = c("NM_005359","NM_000546")
    ipro = getBM(attributes=c("refseq_mrna","interpro","interpro_description"), 
                 filters="refseq_mrna",
                 values=refseqids, 
                 mart=ensembl)
    ipro
    ##    refseq_mrna  interpro                                   interpro_description
    ## 1    NM_000546 IPR002117                           p53 tumour suppressor family
    ## 2    NM_000546 IPR008967             p53-like transcription factor, DNA-binding
    ## 3    NM_000546 IPR010991                            p53, tetramerisation domain
    ## 4    NM_000546 IPR011615                                p53, DNA-binding domain
    ## 5    NM_000546 IPR012346 p53/RUNT-type transcription factor, DNA-binding domain
    ## 6    NM_000546 IPR013872                             p53 transactivation domain
    ## 7    NM_005359 IPR001132                              SMAD domain, Dwarfin-type
    ## 8    NM_005359 IPR003619                           MAD homology 1, Dwarfin-type
    ## 9    NM_005359 IPR008984                                        SMAD/FHA domain
    ## 10   NM_005359 IPR013019                                      MAD homology, MH1
    ## 11   NM_005359 IPR013790                                                Dwarfin
    ## 12   NM_005359 IPR017855                                       SMAD domain-like

    4.5 Select all Affymetrix identifiers on the hgu133plus2 chip and Ensembl gene identifiers for genes located on chromosome 16 between basepair 1100000 and 1250000.

    In this example we will again use multiple filters: chromosome_name, start, and end as we filter on these three conditions. Note that when a chromosome name, a start position and an end position are jointly used as filters, the BioMart webservice interprets this as return everything from the given chromosome between the given start and end positions.

    getBM(attributes = c('affy_hg_u133_plus_2','ensembl_gene_id'), 
          filters = c('chromosome_name','start','end'),
          values = list(16,1100000,1250000), 
          mart = ensembl)
    ##    affy_hg_u133_plus_2 ensembl_gene_id
    ## 1                      ENSG00000260702
    ## 2            215502_at ENSG00000260532
    ## 3                      ENSG00000273551
    ## 4            205845_at ENSG00000196557
    ## 5                      ENSG00000196557
    ## 6                      ENSG00000260403
    ## 7                      ENSG00000259910
    ## 8                      ENSG00000261294
    ## 9          220339_s_at ENSG00000116176
    ## 10                     ENSG00000277010
    ## 11         205683_x_at ENSG00000197253
    ## 12         207134_x_at ENSG00000197253
    ## 13         217023_x_at ENSG00000197253
    ## 14         210084_x_at ENSG00000197253
    ## 15         215382_x_at ENSG00000197253
    ## 16         216474_x_at ENSG00000197253
    ## 17         205683_x_at ENSG00000172236
    ## 18         207134_x_at ENSG00000172236
    ## 19         217023_x_at ENSG00000172236
    ## 20         210084_x_at ENSG00000172236
    ## 21         215382_x_at ENSG00000172236
    ## 22         216474_x_at ENSG00000172236

    4.6 Retrieve all entrezgene identifiers and HUGO gene symbols of genes which have a “MAP kinase activity†GO term associated with it.

    The GO identifier for MAP kinase activity is GO:0004707. In our query we will use go_id as our filter, and entrezgene and hgnc_symbol as attributes. Here’s the query:

    getBM(attributes = c('entrezgene','hgnc_symbol'), 
          filters = 'go', 
          values = 'GO:0004707', 
          mart = ensembl)
    ##    entrezgene hgnc_symbol
    ## 1      225689      MAPK15
    ## 2        5594       MAPK1
    ## 3        5595       MAPK3
    ## 4        6300      MAPK12
    ## 5        5600      MAPK11
    ## 6       51701         NLK
    ## 7        5598       MAPK7
    ## 8        5596       MAPK4
    ## 9        1432      MAPK14
    ## 10       5603      MAPK13
    ## 11       5597       MAPK6
    ## 12       5599       MAPK8
    ## 13       5601       MAPK9
    ## 14       5602      MAPK10

    4.7 Given a set of EntrezGene identifiers, retrieve 100bp upstream promoter sequences

    All sequence related queries to Ensembl are available through the getSequence() wrapper function. getBM() can also be used directly to retrieve sequences but this can get complicated so using getSequence is recommended.

    Sequences can be retrieved using the getSequence() function either starting from chromosomal coordinates or identifiers.
    The chromosome name can be specified using the chromosome argument. The start and end arguments are used to specify start and end positions on the chromosome. The type of sequence returned can be specified by the seqType argument which takes the following values:

    • cdna
    • peptide for protein sequences
    • 3utr for 3’ UTR sequences
    • 5utr for 5’ UTR sequences
    • gene_exon for exon sequences only
    • transcript_exon for transcript specific exonic sequences only
    • transcript_exon_intron gives the full unspliced transcript, that is exons + introns
    • gene_exon_intron gives the exons + introns of a gene
    • coding gives the coding sequence only
    • coding_transcript_flank gives the flanking region of the transcript including the UTRs, this must be accompanied with a given value for the upstream or downstream attribute
    • coding_gene_flank gives the flanking region of the gene including the UTRs, this must be accompanied with a given value for the upstream or downstream attribute
    • transcript_flank gives the flanking region of the transcript exculding the UTRs, this must be accompanied with a given value for the upstream or downstream attribute
    • gene_flank gives the flanking region of the gene excluding the UTRs, this must be accompanied with a given value for the upstream or downstream attribute

    In MySQL mode the getSequence() function is more limited and the sequence that is returned is the 5’ to 3’+ strand of the genomic sequence, given a chromosome, as start and an end position.

    This task requires us to retrieve 100bp upstream promoter sequences from a set of EntrzGene identifiers. The type argument in getSequence() can be thought of as the filter in this query and uses the same input names given by listFilters(). In our query we use entrezgene for the type argument. Next we have to specify which type of sequences we want to retrieve, here we are interested in the sequences of the promoter region, starting right next to the coding start of the gene. Setting the seqType to coding_gene_flank will give us what we need. The upstream argument is used to specify how many bp of upstream sequence we want to retrieve, here we’ll retrieve a rather short sequence of 100bp. Putting this all together in getSequence() gives:

    entrez=c("673","7157","837")
    getSequence(id = entrez, 
                type="entrezgene",
                seqType="coding_gene_flank",
                upstream=100, 
                mart=ensembl) 
    ##                                                                                      coding_gene_flank entrezgene
    ## 1 CCTCCGCCTCCGCCTCCGCCTCCGCCTCCCCCAGCTCTCCGCCTCCCTTCCCCCTCCCCGCCCGACAGCGGCCGCTCGGGCCCCGGCTCTCGGTTATAAG        673
    ## 2 CACGTTTCCGCCCTTTGCAATAAGGAAATACATAGTTTACTTTCATTTTTGACTCTGAGGCTCTTTCCAACGCTGTAAAAAAGGACAGAGGCTGTTCCCT        837
    ## 3 TCCTTCTCTGCAGGCCCAGGTGACCCAGGGTTGGAAGTGTCTCATGCTGGATCCCCACTTTTCCTCTTGCAGCAGCCAGACTGCCTTCCGGGTCACTGCC       7157

    4.8 Retrieve all 5’ UTR sequences of all genes that are located on chromosome 3 between the positions 185,514,033 and 185,535,839

    As described in the provious task getSequence can also use chromosomal coordinates to retrieve sequences of all genes that lie in the given region. We also have to specify which type of identifier we want to retrieve together with the sequences, here we choose for entrezgene identifiers.

    utr5 = getSequence(chromosome=3, start=185514033, end=185535839,
                       type="entrezgene",
                       seqType="5utr", 
                       mart=ensembl)
    utr5
    ##                                                                                                                                             5utr
    ## 1                                                                                                                           Sequence unavailable
    ## 2                                                        TGAGCAAAATCCCACAGTGGAAACTCTTAAGCCTCTGCGAAGTAAATCATTCTTGTGAATGTGACACACGATCTCTCCAGTTTCCAT
    ## 3                                                                                                        ATTCTTGTGAATGTGACACACGATCTCTCCAGTTTCCAT
    ## 4 AGTCCCTAGGGAACTTCCTGTTGTCACCACACCTCTGAGTCGTCTGAGCTCACTGTGAGCAAAATCCCACAGTGGAAACTCTTAAGCCTCTGCGAAGTAAATCATTCTTGTGAATGTGACACACGATCTCTCCAGTTTCCAT
    ##   entrezgene
    ## 1     200879
    ## 2     200879
    ## 3     200879
    ## 4     200879

    4.9 Retrieve protein sequences for a given list of EntrezGene identifiers

    In this task the type argument specifies which type of identifiers we are using. To get an overview of other valid identifier types we refer to the listFilters() function.

    protein = getSequence(id=c(100, 5728),
                          type="entrezgene",
                          seqType="peptide", 
                          mart=ensembl)
    protein
    ##                                                                                                                                                                                                                                                                                                                                                                                                                peptide
    ## 1 MTAIIKEIVSRNKRRYQEDGFDLDLTYIYPNIIAMGFPAERLEGVYRNNIDDVVRFLDSKHKNHYKIYNLCAERHYDTAKFNCRVAQYPFEDHNPPQLELIKPFCEDLDQWLSEDDNHVAAIHCKAGKGRTGVMICAYLLHRGKFLKAQEALDFYGEVRTRDKKGVTIPSQRRYVYYYSYLLKNHLDYRPVALLFHKMMFETIPMFSGGTCNPQFVVCQLKVKIYSSNSGPTRREDKFMYFEFPQPLPVCGDIKVEFFHKQNKMLKKDKMFHFWVNTFFIPGPEETSEKVENGSLCDQEIDSICSIERADNDKEYLVLTLTKNDLDKANKDKANRYFSPNFKVKLYFTKTVEEPSNPEASSSTSVTPDVSDNEPDHYRYSDTTDSDPENEPFDEDQHTQITKV*
    ## 2                                                                                                                                                                                                                                                           ALLFHKMMFETIPMFSGGTCNPQFVVCQLKVKIYSSNSGPTRREDKFMYFEFPQPLPVCGDIKVEFFHKQNKMLKKDKMFHFWVNTFFIPGPEETSEKVENGSLCDQEIDSICSIERADNDKEYLVLTLTKNDLDKANKDKANRYFSPNFKVS*
    ## 3                                                                                                                                                                                                                                                                                                                                                                                                 Sequence unavailable
    ## 4                                                                                                                                                                                                                                                                                                                                                                                                 Sequence unavailable
    ## 5                                         MAQTPAFDKPKVELHVHLDGSIKPETILYYGRRRGIALPANTAEGLLNVIGMDKPLTLPDFLAKFDYYMPAIAGCREAIKRIAYEFVEMKAKEGVVYVEVRYSPHLLANSKVEPIPWNQAEGDLTPDEVVALVGQGLQEGERDFGVKARSILCCMRHQPNWSPKVVELCKKYQQQTVVAIDLAGDETIPGSSLLPGHVQAYQEAVKSGIHRTVHAGEVGSAEVVKEAVDILKTERLGHGYHTLEDQALYNRLRQENMHFEICPWSSYLTGAWKPDTEHAVIRLKNDQANYSLNTDDPLIFKSTLDTDYQMTKRDMGFTEEEFKRLNINAAKSSFLPEDEKRELLDLLYKAYGMPPSASAGQNL*
    ## 6                                                                                                                                                                                                                                                                                                                                         MAQTPAFDKPKVELHVHLDGSIKPETILYYGRRRGIALPANTAEGLLNVIGMDKPLTLPDFLAKFDYYMPAIARL*
    ## 7                                                                 MAQTPAFDKPKVELHVHLDGSIKPETILYYGRRRGIALPANTAEGLLNVIGMDKPLTLPDFLAKFDYYMPAIAGCREAIKRIAYEFVEMKAKEGVVYVEVRYSPHLLANSKVEPIPWNQAEGDLTPDEVVALVGQGLQEGERDFGVKARSILCCMRHQPNWSPKVVELCKKYQQQTVVAIDLAGDETIPGSSLLPGHVQAYQAVDILKTERLGHGYHTLEDQALYNRLRQENMHFEICPWSSYLTGAWKPDTEHAVIRLKNDQANYSLNTDDPLIFKSTLDTDYQMTKRDMGFTEEEFKRLNINAAKSSFLPEDEKRELLDLLYKAYGMPPSASAGQNL*
    ## 8                                                                                                                                             MAQTPAFDKPKVELHVHLDGSIKPETILYYGRRRGIALPANTAEGLLNVIGMDKPLTLPDFLAKFDYYMPAIAGCREAIKRIAYEFVEMKAKEGVVYVEVRYSPHLLANSKVEPIPWNQAEGDLTPDEVVALVGQGLQEGERDFGVKARSILCCMRHQPNWSPKVVELCKKYQQQTVVAIDLAGDETIPGSSLLPGHVQAYQEAVKSGIHRTVHAGEVGSAEVVKEAVDILKTERLGHGYHTLEDQALYNRLRQENMHFEAQK*
    ##   entrezgene
    ## 1       5728
    ## 2       5728
    ## 3       5728
    ## 4        100
    ## 5        100
    ## 6        100
    ## 7        100
    ## 8        100

    4.10 Retrieve known SNPs located on the human chromosome 8 between positions 148350 and 148612

    For this example we’ll first have to connect to a different BioMart database, namely snp.

    snpmart = useMart(biomart = "ENSEMBL_MART_SNP", dataset="hsapiens_snp")

    The listAttributes() and listFilters() functions give us an overview of the available attributes and filters.
    From these we need: refsnp_id, allele, chrom_start and chrom_strand as attributes; and as filters we’ll use: chrom_start, chrom_end and chr_name.
    Note that when a chromosome name, a start position and an end position are jointly used as filters, the BioMart webservice interprets this as return everything from the given chromosome between the given start and end positions. Putting our selected attributes and filters into getBM gives:

    getBM(attributes = c('refsnp_id','allele','chrom_start','chrom_strand'), 
          filters = c('chr_name','start','end'), 
          values = list(8,148350,148612), 
          mart = snpmart)
    ##     refsnp_id allele chrom_start chrom_strand
    ## 1 rs868546642    A/G      148372            1
    ## 2 rs547420070    A/C      148373            1
    ## 3  rs77274555    G/A      148391            1
    ## 4 rs567299969    T/A      148394            1
    ## 5 rs368076569    G/A      148407            1
    ## 6 rs745318437    C/G      148497            1
    ## 7 rs190721891    C/G      148576            1

    4.11 Given the human gene TP53, retrieve the human chromosomal location of this gene and also retrieve the chromosomal location and RefSeq id of its homolog in mouse.

    The getLDS() (Get Linked Dataset) function provides functionality to link 2 BioMart datasets which each other and construct a query over the two datasets. In Ensembl, linking two datasets translates to retrieving homology data across species. The usage of getLDS is very similar to getBM(). The linked dataset is provided by a separate Mart object and one has to specify filters and attributes for the linked dataset. Filters can either be applied to both datasets or to one of the datasets. Use the listFilters and listAttributes functions on both Mart objects to find the filters and attributes for each dataset (species in Ensembl). The attributes and filters of the linked dataset can be specified with the attributesL and filtersL arguments. Entering all this information into getLDS() gives:

    human = useMart("ensembl", dataset = "hsapiens_gene_ensembl")
    mouse = useMart("ensembl", dataset = "mmusculus_gene_ensembl")
    getLDS(attributes = c("hgnc_symbol","chromosome_name", "start_position"),
           filters = "hgnc_symbol", values = "TP53",mart = human,
          attributesL = c("refseq_mrna","chromosome_name","start_position"), martL = mouse)
    ##   HGNC.symbol Chromosome.scaffold.name Gene.start..bp. RefSeq.mRNA.ID Chromosome.scaffold.name.1 Gene.start..bp..1
    ## 1        TP53                       17         7661779                                        11          69580359
    ## 2        TP53                       17         7661779   NM_001127233                         11          69580359
    ## 3        TP53                       17         7661779      NM_011640                         11          69580359

    5 Using archived versions of Ensembl

    It is possible to query archived versions of Ensembl through biomaRt.

    biomaRt provides the function listEnsemblArchives() to view the available archives. This function takes no arguments, and produces a table containing the names of the available archived versions, the date they were first available, and the URL where they can be accessed.

    listEnsemblArchives()
    ##       version          date       url                                 
    ##  [1,] "Ensembl GRCh37" "Feb 2014" "http://grch37.ensembl.org"         
    ##  [2,] "Ensembl 91"     "Dec 2017" "http://Dec2017.archive.ensembl.org"
    ##  [3,] "Ensembl 90"     "Aug 2017" "http://Aug2017.archive.ensembl.org"
    ##  [4,] "Ensembl 89"     "May 2017" "http://May2017.archive.ensembl.org"
    ##  [5,] "Ensembl 88"     "Mar 2017" "http://Mar2017.archive.ensembl.org"
    ##  [6,] "Ensembl 87"     "Dec 2016" "http://Dec2016.archive.ensembl.org"
    ##  [7,] "Ensembl 86"     "Oct 2016" "http://Oct2016.archive.ensembl.org"
    ##  [8,] "Ensembl 85"     "Jul 2016" "http://Jul2016.archive.ensembl.org"
    ##  [9,] "Ensembl 84"     "Mar 2016" "http://Mar2016.archive.ensembl.org"
    ## [10,] "Ensembl 83"     "Dec 2015" "http://Dec2015.archive.ensembl.org"
    ## [11,] "Ensembl 82"     "Sep 2015" "http://Sep2015.archive.ensembl.org"
    ## [12,] "Ensembl 81"     "Jul 2015" "http://Jul2015.archive.ensembl.org"
    ## [13,] "Ensembl 80"     "May 2015" "http://May2015.archive.ensembl.org"
    ## [14,] "Ensembl 79"     "Mar 2015" "http://Mar2015.archive.ensembl.org"
    ## [15,] "Ensembl 78"     "Dec 2014" "http://Dec2014.archive.ensembl.org"
    ## [16,] "Ensembl 77"     "Oct 2014" "http://Oct2014.archive.ensembl.org"
    ## [17,] "Ensembl 76"     "Aug 2014" "http://Aug2014.archive.ensembl.org"
    ## [18,] "Ensembl 75"     "Feb 2014" "http://Feb2014.archive.ensembl.org"
    ## [19,] "Ensembl 74"     "Dec 2013" "http://Dec2013.archive.ensembl.org"
    ## [20,] "Ensembl 67"     "May 2012" "http://May2012.archive.ensembl.org"
    ## [21,] "Ensembl 54"     "May 2009" "http://May2009.archive.ensembl.org"

    Alternatively, one can use the http://www.ensembl.org website to find archived version. From the main page scroll down the bottom of the page, click on ‘view in Archive’ and select the archive you need.

    You will notice that there is an archive URL even for the current release of Ensembl. It can be useful to use this if you wish to ensure that script you write now will return exactly the same results in the future. Using www.ensembl.org will always access the current release, and so the data retrieved may change over time as new releases come out.

    Whichever method you use to find the URL of the archive you wish to query, copy the url and use that in the host argument as shown below to connect to the specified BioMart database. The example below shows how to query Ensembl 54.

    listMarts(host = 'may2009.archive.ensembl.org')
    ##                biomart              version
    ## 1 ENSEMBL_MART_ENSEMBL           Ensembl 54
    ## 2     ENSEMBL_MART_SNP Ensembl Variation 54
    ## 3    ENSEMBL_MART_VEGA              Vega 35
    ## 4             REACTOME   Reactome(CSHL US) 
    ## 5     wormbase_current   WormBase (CSHL US)
    ## 6                pride       PRIDE (EBI UK)
    ensembl54 <- useMart(host='may2009.archive.ensembl.org', 
                         biomart='ENSEMBL_MART_ENSEMBL', 
                         dataset='hsapiens_gene_ensembl')

    6 Using a BioMart other than Ensembl

    To demonstrate the use of the biomaRt package with non-Ensembl databases the next query is performed using the Wormbase ParaSite BioMart. In this example, we use the listMarts() function to find the name of the available marts, given the URL of Wormbase. We use this to connect to Wormbase BioMart, find and select the gene dataset, and print the first 6 available attributes and filters. Then we use a list of gene names as filter and retrieve associated transcript IDs and the transcript biotype.

    listMarts(host = "parasite.wormbase.org")
    ##         biomart       version
    ## 1 parasite_mart ParaSite Mart
    wormbase = useMart(biomart = "parasite_mart", host = "parasite.wormbase.org")
    listDatasets(wormbase)
    ##     dataset         description version
    ## 1 wbps_gene All Species (WBPS9)       9
    wormbase <- useDataset(mart = wormbase, dataset = "wbps_gene")
    head(listFilters(wormbase))
    ##                  name     description
    ## 1     species_id_1010          Genome
    ## 2 nematode_clade_1010  Nematode Clade
    ## 3     chromosome_name Chromosome name
    ## 4               start           Start
    ## 5                 end             End
    ## 6              strand          Strand
    head(listAttributes(wormbase))
    ##                      name        description         page
    ## 1          species_id_key      Internal Name feature_page
    ## 2    production_name_1010     Genome project feature_page
    ## 3       display_name_1010        Genome name feature_page
    ## 4        taxonomy_id_1010        Taxonomy ID feature_page
    ## 5 assembly_accession_1010 Assembly accession feature_page
    ## 6     nematode_clade_1010     Nematode clade feature_page
    getBM(attributes = c("external_gene_id", "wbps_transcript_id", "transcript_biotype"), 
          filters="gene_name", 
          values=c("unc-26","his-33"), 
          mart=wormbase)
    ##   external_gene_id wbps_transcript_id transcript_biotype
    ## 1           his-33           F17E9.13     protein_coding
    ## 2           unc-26            JC8.10a     protein_coding
    ## 3           unc-26            JC8.10b     protein_coding
    ## 4           unc-26          JC8.10c.1     protein_coding
    ## 5           unc-26          JC8.10c.2     protein_coding
    ## 6           unc-26            JC8.10d     protein_coding

    7 biomaRt helper functions

    This section describes a set of biomaRt helper functions that can be used to export FASTA format sequences, retrieve values for certain filters and exploring the available filters and attributes in a more systematic manner.

    7.1 exportFASTA

    The data.frames obtained by the getSequence function can be exported to FASTA files using the exportFASTA() function. One has to specify the data.frame to export and the filename using the file argument.

    7.2 Finding out more information on filters

    7.2.1 filterType

    Boolean filters need a value TRUE or FALSE in biomaRt. Setting the value TRUE will include all information that fulfill the filter requirement. Setting FALSE will exclude the information that fulfills the filter requirement and will return all values that don’t fulfill the filter. For most of the filters, their name indicates if the type is a boolean or not and they will usually start with “withâ€. However this is not a rule and to make sure you got the type right you can use the function filterType() to investigate the type of the filter you want to use.

    filterType("with_affy_hg_u133_plus_2",ensembl)
    ## [1] "boolean_list"

    7.2.2 filterOptions

    Some filters have a limited set of values that can be given to them. To know which values these are one can use the filterOptions() function to retrieve the predetermed values of the respective filter.

    filterOptions("biotype",ensembl)
    ## [1] "[3prime_overlapping_ncRNA,antisense_RNA,bidirectional_promoter_lncRNA,IG_C_gene,IG_C_pseudogene,IG_D_gene,IG_J_gene,IG_J_pseudogene,IG_pseudogene,IG_V_gene,IG_V_pseudogene,lincRNA,macro_lncRNA,miRNA,misc_RNA,Mt_rRNA,Mt_tRNA,non_coding,polymorphic_pseudogene,processed_pseudogene,processed_transcript,protein_coding,pseudogene,ribozyme,rRNA,scaRNA,scRNA,sense_intronic,sense_overlapping,snoRNA,snRNA,sRNA,TEC,transcribed_processed_pseudogene,transcribed_unitary_pseudogene,transcribed_unprocessed_pseudogene,translated_processed_pseudogene,TR_C_gene,TR_D_gene,TR_J_gene,TR_J_pseudogene,TR_V_gene,TR_V_pseudogene,unitary_pseudogene,unprocessed_pseudogene,vaultRNA]"

    If there are no predetermed values e.g. for the entrezgene filter, then filterOptions() will return the type of filter it is. And most of the times the filter name or it’s description will suggest what values one case use for the respective filter (e.g. entrezgene filter will work with enterzgene identifiers as values)

    7.3 Attribute Pages

    For large BioMart databases such as Ensembl, the number of attributes displayed by the listAttributes() function can be very large. In BioMart databases, attributes are put together in pages, such as sequences, features, homologs for Ensembl. An overview of the attributes pages present in the respective BioMart dataset can be obtained with the attributePages() function.

    pages = attributePages(ensembl)
    pages
    ## [1] "feature_page" "structure"    "homologs"     "snp"          "snp_somatic"  "sequences"

    To show us a smaller list of attributes which belong to a specific page, we can now specify this in the listAttributes() function. The set of attributes is still quite long, so we use head() to show only the first few items here.

    head(listAttributes(ensembl, page="feature_page"))
    ##                            name                  description         page
    ## 1               ensembl_gene_id               Gene stable ID feature_page
    ## 2       ensembl_gene_id_version       Gene stable ID version feature_page
    ## 3         ensembl_transcript_id         Transcript stable ID feature_page
    ## 4 ensembl_transcript_id_version Transcript stable ID version feature_page
    ## 5            ensembl_peptide_id            Protein stable ID feature_page
    ## 6    ensembl_peptide_id_version    Protein stable ID version feature_page

    We now get a short list of attributes related to the region where the genes are located.

    8 Local BioMart databases

    The biomaRt package can be used with a local install of a public BioMart database or a locally developed BioMart database and web service. In order for biomaRt to recognize the database as a BioMart, make sure that the local database you create has a name conform with database_mart_version where database is the name of the database and version is a version number. No more underscores than the ones showed should be present in this name. A possible name is for example ensemblLocal_mart_46. ## Minimum requirements for local database installation More information on installing a local copy of a BioMart database or develop your own BioMart database and webservice can be found on http://www.biomart.org Once the local database is installed you can use biomaRt on this database by:

    listMarts(host="www.myLocalHost.org", path="/myPathToWebservice/martservice")
    mart=useMart("nameOfMyMart",dataset="nameOfMyDataset",host="www.myLocalHost.org", path="/myPathToWebservice/martservice")

    For more information on how to install a public BioMart database see: http://www.biomart.org/install.html and follow link databases.

    9 Using select()

    In order to provide a more consistent interface to all annotations in Bioconductor the select(), columns(), keytypes() and keys() have been implemented to wrap some of the existing functionality above. These methods can be called in the same manner that they are used in other parts of the project except that instead of taking a AnnotationDb derived class they take instead a Mart derived class as their 1st argument. Otherwise usage should be essentially the same. You still use columns() to discover things that can be extracted from a Mart, and keytypes() to discover which things can be used as keys with select().

    mart <- useMart(dataset="hsapiens_gene_ensembl",biomart='ensembl')
    head(keytypes(mart), n=3)
    ## [1] "affy_hc_g110"        "affy_hg_focus"       "affy_hg_u133_plus_2"
    head(columns(mart), n=3)
    ## [1] "3_utr_end"   "3_utr_end"   "3_utr_start"

    And you still can use keys() to extract potential keys, for a particular key type.

    k = keys(mart, keytype="chromosome_name")
    head(k, n=3)
    ## [1] "1" "2" "3"

    When using keys(), you can even take advantage of the extra arguments that are available for others keys methods.

    k = keys(mart, keytype="chromosome_name", pattern="LRG")
    head(k, n=3)
    ## character(0)

    Unfortunately the keys() method will not work with all key types because they are not all supported.

    But you can still use select() here to extract columns of data that match a particular set of keys (this is basically a wrapper for getBM()).

    affy=c("202763_at","209310_s_at","207500_at")
    select(mart, keys=affy, columns=c('affy_hg_u133_plus_2','entrezgene'),
      keytype='affy_hg_u133_plus_2')
    ##   affy_hg_u133_plus_2 entrezgene
    ## 1           202763_at        836
    ## 2         209310_s_at        837
    ## 3           207500_at        838

    So why would we want to do this when we already have functions like getBM()? For two reasons: 1) for people who are familiar with select and it’s helper methods, they can now proceed to use biomaRt making the same kinds of calls that are already familiar to them and 2) because the select method is implemented in many places elsewhere, the fact that these methods are shared allows for more convenient programmatic access of all these resources. An example of a package that takes advantage of this is the OrganismDbi package. Where several packages can be accessed as if they were one resource.

    10 Session Info

    sessionInfo()
    ## R version 3.4.3 (2017-11-30)
    ## Platform: x86_64-pc-linux-gnu (64-bit)
    ## Running under: Ubuntu 16.04.3 LTS
    ## 
    ## Matrix products: default
    ## BLAS: /home/biocbuild/bbs-3.6-bioc/R/lib/libRblas.so
    ## LAPACK: /home/biocbuild/bbs-3.6-bioc/R/lib/libRlapack.so
    ## 
    ## locale:
    ##  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_US.UTF-8        LC_COLLATE=C              
    ##  [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8    LC_PAPER=en_US.UTF-8       LC_NAME=C                 
    ##  [9] LC_ADDRESS=C               LC_TELEPHONE=C             LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       
    ## 
    ## attached base packages:
    ## [1] stats     graphics  grDevices utils     datasets  methods   base     
    ## 
    ## other attached packages:
    ## [1] biomaRt_2.34.1  BiocStyle_2.6.1
    ## 
    ## loaded via a namespace (and not attached):
    ##  [1] Rcpp_0.12.14         AnnotationDbi_1.40.0 knitr_1.17           magrittr_1.5         progress_1.1.2      
    ##  [6] IRanges_2.12.0       BiocGenerics_0.24.0  bit_1.1-12           R6_2.2.2             rlang_0.1.4         
    ## [11] httr_1.3.1           stringr_1.2.0        blob_1.1.0           tools_3.4.3          parallel_3.4.3      
    ## [16] Biobase_2.38.0       DBI_0.7              htmltools_0.3.6      assertthat_0.2.0     yaml_2.1.16         
    ## [21] bit64_0.9-7          rprojroot_1.2        digest_0.6.13        tibble_1.3.4         bookdown_0.5        
    ## [26] S4Vectors_0.16.0     bitops_1.0-6         curl_3.1             RCurl_1.95-4.8       memoise_1.1.0       
    ## [31] evaluate_0.10.1      RSQLite_2.0          rmarkdown_1.8        stringi_1.1.6        compiler_3.4.3      
    ## [36] prettyunits_1.0.2    backports_1.1.2      stats4_3.4.3         XML_3.98-1.9
    warnings()
    ## NULL
    biomaRt/inst/scripts/0000755000175400017540000000000013175713423015647 5ustar00biocbuildbiocbuildbiomaRt/inst/scripts/Integration-NP.R0000644000175400017540000001373213175713423020576 0ustar00biocbuildbiocbuild###-------------------------------------------------- ### Step 1 ###-------------------------------------------------- library('biomaRt') library('affy') library('gplots') library('lattice') ###-------------------------------------------------- ### Step 2 ###-------------------------------------------------- sampleAnnot = read.AnnotatedDataFrame('E-TABM-157.txt',row.names = 'Array.Data.File') mRNAraw = ReadAffy(phenoData = sampleAnnot, sampleNames = sampleAnnot$Source.Name) ###-------------------------------------------------- ### Step 3 ###-------------------------------------------------- mRNA = rma(mRNAraw) ###-------------------------------------------------- ### Step 4 ###-------------------------------------------------- probesetvar = apply(exprs(mRNA), 1, var) ord = order(probesetvar, decreasing=TRUE)[1:200] pca = prcomp(t(exprs(mRNA)[ord,]), scale=TRUE) typeColors = c('Lu'='firebrick1','BaA'='dodgerblue2','BaB'='darkblue') plot(pca$x[, 1:2], pch=16, col=typeColors[as.character(mRNA$CancerType)], xlab='PC1', ylab='PC2', asp=1) legend('topleft', c('luminal', 'basal A', 'basal B'), fill=typeColors) ###-------------------------------------------------- ### Step 5 ###-------------------------------------------------- cghData = read.csv('aCGH.csv', header=TRUE, row.names=1) cgh = new('ExpressionSet', exprs = as.matrix(cghData[,4:56])) featureData(cgh) = as(cghData[,1:3], 'AnnotatedDataFrame') ###-------------------------------------------------- ### Step 6 ###-------------------------------------------------- chr1 = which(featureData(cgh)$Chrom == 1) clColors = c('MCF10A' = 'dodgerblue3', 'BT483' = 'orange', 'BT549' = 'olivedrab') logRatio = exprs(cgh)[chr1, names(clColors)] kb = rep(featureData(cgh)$kb[chr1], times=ncol(logRatio)) clName = rep(names(clColors), each=nrow(logRatio)) print(xyplot( logRatio ~ kb | clName, pch=16, layout=c(1,3), ylim=c(-0.5, 1.1), panel=function(...){ panel.xyplot(...,col=clColors[panel.number()]) panel.abline(h=0, col='firebrick2') })) ###-------------------------------------------------- ### Step 7 ###-------------------------------------------------- ensembl = useMart('ensembl', dataset='hsapiens_gene_ensembl') probes = getBM(attributes = c('affy_hg_u133a', 'start_position'), filters = c('chromosome_name', 'with_affy_hg_u133a'), values = list(1, TRUE), mart = ensembl) ###-------------------------------------------------- ### Step 8 ###-------------------------------------------------- xpr = exprs(mRNA)[probes[,'affy_hg_u133a'], c('BT549','BT483')] pos = probes[,'start_position'] plot(xpr, pch=16, cex=0.5, col=ifelse(pos>140e6, 'red', 'darkgrey')) ###-------------------------------------------------- ### Step 9 ###-------------------------------------------------- logRatios = xpr[,2] - xpr[,1] t.test(logRatios ~ (pos>140e6)) ###-------------------------------------------------- ### Step 10 ###-------------------------------------------------- proteinData = read.csv('mmc6.csv', header=TRUE, row.names=1) protein = new('ExpressionSet', exprs = as.matrix(proteinData)) ###-------------------------------------------------- ### Step 11 ###-------------------------------------------------- prmax = apply(exprs(protein), 1, max) barplot(prmax, las=2) ###-------------------------------------------------- ### Step 12 ###-------------------------------------------------- hmColors = colorRampPalette(c('white', 'darkblue'))(256) sideColors = typeColors[as.character(pData(mRNA)[ sampleNames(protein), 'CancerType'])] sideColors[is.na(sideColors)] = 'grey' heatmap.2(exprs(protein)/prmax, col=hmColors, trace='none', ColSideColors = sideColors) ###-------------------------------------------------- ### Step 13 ###-------------------------------------------------- samples = intersect(sampleNames(protein), sampleNames(mRNA)) samples = intersect(samples, sampleNames(cgh)) mRNA = mRNA[,samples] protein = protein[,samples] cgh = cgh[,samples] ###-------------------------------------------------- ### Step 14 ###-------------------------------------------------- map = getBM(attributes = c('ensembl_gene_id', 'affy_hg_u133a', 'hgnc_symbol'), filters = c('hgnc_symbol', 'with_hgnc', 'with_affy_hg_u133a'), values = list(featureNames(protein), TRUE, TRUE), mart = ensembl) head(map) ###-------------------------------------------------- ### Step 15 ###-------------------------------------------------- geneProbesetsAll = split(map$affy_hg_u133a, map$hgnc_symbol) table(listLen(geneProbesetsAll)) ###-------------------------------------------------- ### Step 16 ###-------------------------------------------------- geneProbesets = lapply(geneProbesetsAll, function(x) { good = grep('[0-9]._at', x) if (length(good)>0) x[good] else x }) summaries = lapply(geneProbesets, function(i) { colMeans(exprs(mRNA)[i,,drop=FALSE]) }) summarized_mRNA = do.call(rbind, summaries) ###-------------------------------------------------- ### Step 17 ###-------------------------------------------------- colors = c('orange','olivedrab') correlation = cor(summarized_mRNA['AURKA',], exprs(protein)['AURKA',], method='spearman') matplot(cbind( summarized_mRNA['AURKA', ], log2(exprs(protein)['AURKA', ])), type='l', col=colors, lwd=2, lty=1, ylab='mRNA and protein expression levels', xlab='cell lines', main=bquote(rho==.(round(correlation, 3)))) legend('bottomright', c('mRNA','protein'), fill=colors) ###-------------------------------------------------- ### Step 18 ###-------------------------------------------------- samples = sampleNames(protein)[c(5,11,16,24)] v_mRNA = as.vector(summarized_mRNA[,samples]) v_protein = as.vector(exprs(protein)[ rownames(summarized_mRNA), samples]) print(xyplot( v_protein ~ v_mRNA | rep(samples, each = nrow(summarized_mRNA)), pch=16, xlab = 'mRNA level', scales=list(y=list(log=TRUE)), ylab='protein level')) biomaRt/man/0000755000175400017540000000000013175713423013756 5ustar00biocbuildbiocbuildbiomaRt/man/Mart-class.Rd0000644000175400017540000000043513175713423016255 0ustar00biocbuildbiocbuild\name{Mart-class} \alias{Mart-class} \alias{show,Mart-method} \title{Class Mart} \description{Represents a Mart class, containing connections to different BioMarts} \section{Methods}{ \describe{\code{show} Print summary of the object} } \author{Steffen Durinck} \keyword{methods} biomaRt/man/NP2009code.Rd0000644000175400017540000000104313175713423015726 0ustar00biocbuildbiocbuild\name{NP2009code} \alias{NP2009code} \title{Display the analysis code from the 2009 Nature protocols paper} \description{This function opens an editor displaying the analysis code of the Nature Protocols 2009 paper} \usage{NP2009code()} \details{The \code{\link{edit}} function uses \code{getOption("editor")} to select the editor. Use, for instance, \code{options(editor="emacs")} to set another editor.} \author{Steffen Durinck, Wolfgang Huber} \seealso{\code{\link{edit}}} \examples{ if(interactive()){ NP2009code() } } \keyword{methods} biomaRt/man/attributePages.Rd0000644000175400017540000000125013175713423017226 0ustar00biocbuildbiocbuild\name{attributePages} \alias{attributePages} \title{Gives a summary of the attribute pages} \description{Attributes in BioMart databases are grouped together in attribute pages. The attributePages function gives a summary of the attribute categories and groups present in the BioMart. These page names can be used to display only a subset of the available attributes in the listAttributes function.} \usage{attributePages(mart)} \arguments{ \item{mart}{object of class Mart, created with the useMart function.} } \author{Steffen Durinck} \examples{ if(interactive()){ mart = useMart("ensembl", dataset="hsapiens_gene_ensembl") attributeSummary(mart) } } \keyword{methods} biomaRt/man/exportFASTA.Rd0000644000175400017540000000117213175713423016346 0ustar00biocbuildbiocbuild\name{exportFASTA} \alias{exportFASTA} \title{Exports getSequence results to FASTA format} \description{Exports getSequence results to FASTA format} \usage{exportFASTA(sequences, file)} \arguments{ \item{sequences}{A data.frame that was the output of the getSequence function} \item{file}{File to which you want to write the data} } \author{Steffen Durinck} \examples{ if(interactive()){ mart <- useMart("ensembl", dataset="hsapiens_gene_ensembl") #seq<-getSequence(chromosome=c(2,2),start=c(100000,30000),end=c(100300,30500),mart=mart) #exportFASTA(seq,file="test.fasta") martDisconnect(mart = mart) } } \keyword{methods} biomaRt/man/filterOptions.Rd0000644000175400017540000000077213175713423017114 0ustar00biocbuildbiocbuild\name{filterOptions} \alias{filterOptions} \title{Displays the filter options} \description{Displays a set of predetermed values for the specified filter (if available).} \usage{filterOptions(filter,mart)} \arguments{ \item{filter}{A valid filter name.} \item{mart}{object of class Mar created using the useMart functiont} } \author{Steffen Durinck} \examples{ if(interactive()){ mart = useMart("ensembl", dataset="hsapiens_gene_ensembl") filterOptions("chromosome_name", mart) } } \keyword{methods} biomaRt/man/filterType.Rd0000644000175400017540000000100613175713423016371 0ustar00biocbuildbiocbuild\name{filterType} \alias{filterType} \title{Displays the filter type} \description{Displays the type of the filer given a filter name.} \usage{filterType(filter,mart)} \arguments{ \item{filter}{A valid filter name. Valid filters are given by the listFilters function} \item{mart}{object of class Mart, created using the useMart function} } \author{Steffen Durinck} \examples{ if(interactive()){ mart = useMart("ensembl", dataset="hsapiens_gene_ensembl") filterType("chromosome_name", mart) } } \keyword{methods} biomaRt/man/getBM.Rd0000644000175400017540000000562113175713423015247 0ustar00biocbuildbiocbuild\name{getBM} \alias{getBM} \title{Retrieves information from the BioMart database} \description{This function is the main biomaRt query function. Given a set of filters and corresponding values, it retrieves the user specified attributes from the BioMart database one is connected to.} \usage{getBM(attributes, filters = "", values = "", mart, curl = NULL, checkFilters = TRUE, verbose = FALSE, uniqueRows = TRUE, bmHeader = FALSE, quote = "\"")} \value{A \code{data.frame}. There is no implicit mapping between its rows and the function arguments (e.g. \code{filters}, \code{values}), therefore make sure to have the relevant identifier(s) returned by specifying them in \code{attributes}. See Examples.} \arguments{ \item{attributes}{Attributes you want to retrieve. A possible list of attributes can be retrieved using the function listAttributes.} \item{filters}{Filters (one or more) that should be used in the query. A possible list of filters can be retrieved using the function listFilters.} \item{values}{Values of the filter, e.g. vector of affy IDs. If multiple filters are specified then the argument should be a list of vectors of which the position of each vector corresponds to the position of the filters in the filters argument.} \item{mart}{object of class Mart, created with the useMart function.} \item{curl}{An optional 'CURLHandle' object, that can be used to speed up getBM when used in a loop.} \item{checkFilters}{Sometimes attributes where a value needs to be specified, for example upstream\_flank with value 20 for obtaining upstream sequence flank regions of length 20bp, are treated as filters in BioMarts. To enable such a query to work, one must specify the attribute as a filter and set checkFilters = FALSE for the query to work.} \item{verbose}{When using biomaRt in webservice mode and setting verbose to TRUE, the XML query to the webservice will be printed.} \item{uniqueRows}{If the result of a query contains multiple identical rows, setting this argument to TRUE (default) will result in deleting the duplicated rows in the query result at the server side.} \item{bmHeader}{Boolean to indicate if the result retrieved from the BioMart server should include the data headers or not, defaults to FALSE. This should only be switched on if the default behavior results in errors, setting to on might still be able to retrieve your data in that case} \item{quote}{Sometimes parsing of the results fails due to errors in the Ensembl data fields such as containing a quote, in such cases you can try to change the value of quote to try to still parse the results.} } \author{Steffen Durinck} \examples{ mart <- useMart(biomart = "ensembl", dataset = "hsapiens_gene_ensembl") getBM(attributes = c("affy_hg_u95av2", "hgnc_symbol", "chromosome_name", "band"), filters = "affy_hg_u95av2", values = c("1939_at","1503_at","1454_at"), mart = mart) } \keyword{methods} biomaRt/man/getBMlist.Rd0000644000175400017540000000300513175713423016135 0ustar00biocbuildbiocbuild\name{getBMlist} \alias{getBMlist} \title{Retrieves information from the BioMart database} \description{This function is the main biomaRt query function. Given a set of filters and corresponding values, it retrieves the user specified attributes from the BioMart database one is connected to} \usage{getBMlist(attributes, filters = "", values = "", mart, list.names = NULL, na.value = NA, verbose = FALSE, giveWarning = TRUE)} \arguments{ \item{attributes}{Attributes you want to retrieve. A possible list of attributes can be retrieved using the function listAttributes.} \item{filters}{Filters (one or more) that should be used in the query. A possible list of filters can be retrieved using the function listFilters.} \item{values}{Values of the filter, e.g. vector of affy IDs. If multiple filters are specified then the argument should be a list of vectors of which the position of each vector corresponds to the position of the filters in the filters argument.} \item{mart}{object of class Mart, created with the useMart function.} \item{list.names}{names for objects in list} \item{na.value}{value to give when result is empty} \item{verbose}{When using biomaRt in webservice mode and setting verbose to TRUE, the XML query to the webservice will be printed.} \item{giveWarning}{Gives a warning about best practices of biomaRt and recommends using getBM instead of getBMlist} } \author{Steffen Durinck} \examples{ if(interactive()){ mart <- useMart("ensembl") datasets <- listDatasets(mart) } } \keyword{methods} biomaRt/man/getGene.Rd0000644000175400017540000000235513175713423015630 0ustar00biocbuildbiocbuild\name{getGene} \alias{getGene} \title{Retrieves gene annotation information given a vector of identifiers} \description{This function retrieves gene annotations from Ensembl given a vector of identifiers. Annotation includes chromsome name, band, start position, end position, gene description and gene symbol. A wide variety of identifiers is available in Ensembl, these can be found with the listFilters function.} \usage{getGene( id, type, mart)} \arguments{ \item{id}{vector of gene identifiers one wants to annotate} \item{type}{type of identifier, possible values can be obtained by the listFilters function. Examples are entrezgene, hgnc\_symbol (for hugo gene symbol), ensembl\_gene\_id, unigene, agilentprobe, affy\_hg\_u133\_plus\_2, refseq\_dna, etc.} \item{mart}{object of class Mart, containing connections to the BioMart databases. You can create such an object using the function useMart.} } \author{Steffen Durinck} \examples{ if(interactive()){ mart = useMart("ensembl", dataset="hsapiens_gene_ensembl") #example using affy id g = getGene( id = "1939_at", type = "affy_hg_u95av2", mart = mart) show(g) #example using Entrez Gene id g = getGene( id = "100", type = "entrezgene", mart = mart) show(g) } } \keyword{methods} biomaRt/man/getLDS.Rd0000644000175400017540000000431713175713423015374 0ustar00biocbuildbiocbuild\name{getLDS} \alias{getLDS} \title{Retrieves information from two linked datasets} \description{This function is the main biomaRt query function that links 2 datasets and retrieves information from these linked BioMart datasets. In Ensembl this translates to homology mapping.} \usage{getLDS(attributes, filters = "", values = "", mart, attributesL, filtersL = "", valuesL = "", martL, verbose = FALSE, uniqueRows = TRUE, bmHeader=TRUE)} \arguments{ \item{attributes}{Attributes you want to retrieve of primary dataset. A possible list of attributes can be retrieved using the function listAttributes.} \item{filters}{Filters that should be used in the query. These filters will be applied to primary dataset. A possible list of filters can be retrieved using the function listFilters.} \item{values}{Values of the filter, e.g. list of affy IDs} \item{mart}{object of class Mart created with the useMart function.} \item{attributesL}{Attributes of linked dataset that needs to be retrieved} \item{filtersL}{Filters to be applied to the linked dataset} \item{valuesL}{Values for the linked dataset filters} \item{martL}{Mart object representing linked dataset} \item{verbose}{When using biomaRt in webservice mode and setting verbose to TRUE, the XML query to the webservice will be printed. Alternatively in MySQL mode the MySQL query will be printed.} \item{uniqueRows}{Logical to indicate if the BioMart web service should return unique rows only or not. Has the value of either TRUE or FALSE} \item{bmHeader}{Boolean to indicate if the result retrieved from the BioMart server should include the data headers or not, defaults to TRUE. This should only be switched off if the default behavior results in errors, setting to off might still be able to retrieve your data in that case} } \author{Steffen Durinck} \examples{ if(interactive()){ human = useMart("ensembl", dataset = "hsapiens_gene_ensembl") mouse = useMart("ensembl", dataset = "mmusculus_gene_ensembl") getLDS(attributes = c("hgnc_symbol","chromosome_name", "start_position"), filters = "hgnc_symbol", values = "TP53", mart = human, attributesL = c("chromosome_name","start_position"), martL = mouse) } } \keyword{methods} biomaRt/man/getSequence.Rd0000644000175400017540000000674313175713423016527 0ustar00biocbuildbiocbuild\name{getSequence} \alias{getSequence} \title{Retrieves sequences} \description{This function retrieves sequences given the chomosome, start and end position or a list of identifiers. Using getSequence in web service mode (default) generates 5' to 3' sequences of the requested type on the correct strand.} \details{The type of sequence returned can be specified by the seqType argument which takes the following values: \itemize{ \item{'cdna': for nucleotide sequences} \item{'peptide': for protein sequences} \item{'3utr': for 3' UTR sequences} \item{'5utr': for 5' UTR sequences} \item{'gene_exon': for exon sequences only} \item{'transcript_exon_intron': gives the full unspliced transcript, that is exons + introns} \item{'gene_exon_intron' gives the exons + introns of a gene;'coding' gives the coding sequence only} \item{'coding_transcript_flank': gives the flanking region of the transcript including the UTRs, this must be accompanied with a given value for the upstream or downstream attribute} \item{'coding_gene_flank': gives the flanking region of the gene including the UTRs, this must be accompanied with a given value for the upstream or downstream attribute} \item{'transcript_flank': gives the flanking region of the transcript exculding the UTRs, this must be accompanied with a given value for the upstream or downstream attribute} \item{'gene_flank': gives the flanking region of the gene excluding the UTRs, this must be accompanied with a given value for the upstream or downstream attribute} } In MySQL mode the getSequence function is more limited and the sequence that is returned is the 5' to 3'+ strand of the genomic sequence, given a chromosome, as start and an end position. So if the sequence of interest is the minus strand, one has to compute the reverse complement of the retrieved sequence, which can be done using functions provided in the matchprobes package. The biomaRt vignette contains more examples on how to use this function. } \usage{getSequence(chromosome, start, end, id, type, seqType, upstream, downstream, mart, verbose = FALSE)} \arguments{ \item{chromosome}{Chromosome name} \item{start}{start position of sequence on chromosome} \item{end}{end position of sequence on chromosome} \item{id}{ An identifier or vector of identifiers.} \item{type}{The type of identifier used. Supported types are hugo, ensembl, embl, entrezgene, refseq, ensemblTrans and unigene. Alternatively one can also use a filter to specify the type. Possible filters are given by the listFilters function} \item{seqType}{Type of sequence that you want to retrieve. Allowed seqTypes are given in the details section.} \item{upstream}{To add the upstream sequence of a specified number of basepairs to the output.} \item{downstream}{To add the downstream sequence of a specified number of basepairs to the output.} \item{mart}{object of class Mart created using the useMart function} \item{verbose}{If verbose = TRUE then the XML query that was send to the webservice will be displayed.} } \author{Steffen Durinck} \examples{ if(interactive()){ mart <- useMart("ensembl", dataset="hsapiens_gene_ensembl") seq = getSequence(id = "BRCA1", type = "hgnc_symbol", seqType = "peptide", mart = mart) show(seq) seq = getSequence(id="1939_at", type="affy_hg_u95av2", seqType="gene_flank", upstream = 20, mart = mart) show(seq) } } \keyword{methods} biomaRt/man/getXML.Rd0000644000175400017540000000216513175713423015411 0ustar00biocbuildbiocbuild\name{getXML} \alias{getXML} \title{Retrieves information from the BioMart database using an XML query} \description{This function is a low level query function bypassing lots of biomaRts internal controls. It allows for a direct XML query to a known BioMart webservice host.} \usage{getXML(host="http://www.biomart.org/biomart/martservice?", xmlquery)} \arguments{ \item{host}{URL to BioMart webservice, is set to http://www.biomart.org/biomart/martservice? by default} \item{xmlquery}{XML query that needs to be send to the webservice} } \author{Steffen Durinck} \examples{ if(interactive()){ xmlquery=" " getXML(host = "www.ensembl.org/biomart/martservice?", xmlquery = xmlquery) } } \keyword{methods} biomaRt/man/listAttributes.Rd0000644000175400017540000000176613175713423017301 0ustar00biocbuildbiocbuild\name{listAttributes} \alias{listAttributes} \title{lists the attributes available in the selected dataset} \description{Attributes are the outputs of a biomaRt query, they are the information we want to retrieve. For example if we want to retrieve all entrez gene identifiers of genes located on chromosome X, entrezgene will be the attribute we use in the query. The listAttributes function lists the available attributes in the selected dataset} \usage{listAttributes(mart, page,what = c("name","description","page"))} \arguments{ \item{mart}{object of class Mart created using the useMart function} \item{page}{Show only the attributes that belong to the specified attribute page.} \item{what}{vector of types of information about the attributes that need to be displayed. Can have values like name, description, fullDescription, page} } \author{Steffen Durinck} \examples{ if(interactive()){ ensembl = useMart("ensembl", dataset="hsapiens_gene_ensembl") listAttributes(ensembl) } } \keyword{methods} biomaRt/man/listDatasets.Rd0000644000175400017540000000116413175713423016713 0ustar00biocbuildbiocbuild\name{listDatasets} \alias{listDatasets} \title{lists the datasets available in the selected BioMart database} \description{Lists the datasets available in the selected BioMart database} \usage{listDatasets(mart, verbose = FALSE)} \arguments{ \item{mart}{object of class Mart created with the useMart function} \item{verbose}{Give detailed output of what the method is doing, for debugging purposes} } \author{Steffen Durinck} \examples{ if(interactive()){ #marts <- listMarts() #index<-grep("ensembl",marts) #mart <- useMart(marts[index]) #listDatasets(mart = mart) #martDisconnect(mart = mart) } } \keyword{methods} biomaRt/man/listEnsembl.Rd0000644000175400017540000000233113175713423016525 0ustar00biocbuildbiocbuild\name{listEnsembl} \alias{listEnsembl} \title{lists the available BioMart databases hosted by Ensembl} \description{This function returns a list of BioMart databases hosted by Ensembl. To establish a connection use the useMart function.} \usage{listEnsembl(mart = NULL,host = "www.ensembl.org", version = NULL, GRCh = NULL, mirror = NULL, verbose = FALSE)} \arguments{ \item{mart}{mart object created with the useEnsembl function. This is optional, as you usually use \code{\link{listMarts}} to see which marts there are to connect to.} \item{host}{Host to connect to if different then www.ensembl.org} \item{version}{Ensembl version to connect to when wanting to connect to an archived Ensembl version} \item{GRCh}{GRCh version to connect to if not the current GRCh38, currently this can only be 37} \item{mirror}{Specify an Ensembl mirror to connect to. The valid options here are 'www', 'uswest', 'useast', 'asia'. If no mirror is specified the default will be used which redirects you to the closest mirror to your location.} \item{verbose}{Give detailed output of what the method is doing, for debugging purposes} } \author{Steffen Durinck} \examples{ if(interactive()){ listEnsembl() } } \keyword{methods} biomaRt/man/listEnsemblArchives.Rd0000644000175400017540000000061013175713423020210 0ustar00biocbuildbiocbuild\name{listEnsemblArchives} \alias{listEnsemblArchives} \title{Lists the available archived versions of Ensembl} \description{Returns a table containing the available archived versions of Ensembl, along with the dates they were created and the URL used to access them.} \usage{listEnsemblArchives()} \arguments{} \author{Mike Smith} \examples{ listEnsemblArchives() } \keyword{methods} biomaRt/man/listFilters.Rd0000644000175400017540000000164713175713423016561 0ustar00biocbuildbiocbuild\name{listFilters} \alias{listFilters} \title{lists the filters available in the selected dataset} \description{Filters are what we use as inputs for a biomaRt query. For example, if we want to retrieve all entrezgene identifiers on chromosome X, \code{chromosome} will be the filter, with corresponding value X.} \usage{listFilters(mart, what = c("name", "description"))} \arguments{ \item{mart}{object of class \code{Mart} created using the \code{\link{useMart}} function} \item{what}{character vector indicating what information to display about the available filters. Valid values are \code{name}, \code{description}, \code{options}, \code{fullDescription}, \code{filters}, \code{type}, \code{operation}, \code{filters8}, \code{filters9}.} } \author{Steffen Durinck} \examples{ if(interactive()){ mart = useMart("ensembl", dataset="hsapiens_gene_ensembl") listFilters(mart) } } \keyword{methods} biomaRt/man/listMarts.Rd0000644000175400017540000000511413175713423016230 0ustar00biocbuildbiocbuild\name{listMarts} \alias{listMarts} \title{lists the available BioMart databases} \description{This function returns a list of BioMart databases to which biomaRt can connect to. By default all public BioMart databases are displayed. To establish a connection use the useMart function.} \usage{listMarts(mart = NULL, host="www.ensembl.org", path="/biomart/martservice", port=80, includeHosts = FALSE, archive = FALSE, ssl.verifypeer = TRUE, ensemblRedirect = TRUE, verbose = FALSE)} \arguments{ \item{mart}{mart object created with the \code{\link{useMart}} function. This is optional, as you usually use \code{\link{listMarts}} to see which marts there are to connect to.} \item{host}{Host to connect to. Defaults to \code{www.ensembl.org}} \item{path}{path to martservice that should be pasted behind the host to get to web service URL} \item{port}{port to use in HTTP communication} \item{includeHosts}{boolean to indicate if function should return host of the BioMart databases} \item{archive}{Boolean to indicate if you want to access archived versions of BioMart database. Note that this argument is now deprecated and will be removed in the future. A better alternative is to specify the url of the archived BioMart you want to access. For Ensembl you can view the list of archives using \code{\link{listEnsemblArchives}}} \item{ssl.verifypeer}{Set SSL peer verification on or off. By default ssl.verifypeer is set to TRUE} \item{ensemblRedirect}{By default when you access Ensembl BioMart it will redirect you to your local mirror, even if you have set a region specific mirror in the \code{host} argument. By setting this argument to \code{FALSE} you can override this behaviour and force access to your specified \code{host}. Defaults to \code{TRUE} to prevent overwhelming the main Ensembl site. If you are accessing a BioMart instance other than Ensembl this should have no effect.} \item{verbose}{Give detailed output of what the method is doing, for debugging purposes} } \details{ If you receive an error message saying 'Unexpected format to the list of available marts', this is often because there is a problem with the BioMart server you are trying to connect to, and something other than the list of available marts is being returned - often some like a 'down for maintainance' page. If you browse to the provided URL and find a page that starts with '\code{}' this is the correct listing and you should report the issue on the Bioconductor support site: https://support.bioconductor.org } \author{Steffen Durinck} \examples{ if(interactive()){ listMarts() } } \keyword{methods} biomaRt/man/select.Rd0000644000175400017540000000736213175713423015534 0ustar00biocbuildbiocbuild%% actual page name \alias{select-methods} \name{select-methods} %% for methods \alias{keys,Mart-method} \alias{columns,Mart-method} \alias{keytypes,Mart-method} \alias{select,Mart-method} %% for convenience \alias{keys} \alias{columns} \alias{keytypes} \alias{select} \title{Retrieve information from the BioMart databases} \description{ \code{select}, \code{columns} and \code{keys} are used together to extract data from a \code{Mart} object. These functions work much the same as the classic biomaRt functions such as \code{getBM} etc. and are provide here to make this easier for people who are comfortable using these methods from other Annotation packages. Examples of other objects in other packages where you can use these methods include (but are not limited to): \code{ChipDb}, \code{OrgDb} \code{GODb}, \code{InparanoidDb} and \code{ReactomeDb}. \code{columns} shows which kinds of data can be returned from the \code{Mart} object. \code{keytypes} allows the user to discover which keytypes can be passed in to \code{select} or \code{keys} as the \code{keytype} argument. \code{keys} returns keys from the \code{Mart} of the type specified by it's \code{keytype} argument. \code{select} is meant to be used with these other methods and has arguments that take the kinds of values that these other methods return. \code{select} will retrieve the results as a data.frame based on parameters for selected \code{keys} and \code{columns} and \code{keytype} arguments. } \usage{ columns(x) keytypes(x) keys(x, keytype, ...) select(x, keys, columns, keytype, ...) } \arguments{ \item{x}{the \code{Mart} object. The dataset of the \code{Mart} object must already be specified for all of these methods.} \item{keys}{the keys to select records for from the database. Keys for some keytypes can be extracted by using the \code{keys} method.} \item{columns}{the columns or kinds of things that can be retrieved from the database. As with \code{keys}, all possible columns are returned by using the \code{columns} method.} \item{keytype}{the keytype that matches the keys used. For the \code{select} methods, this is used to indicate the kind of ID being used with the keys argument. For the \code{keys} method this is used to indicate which kind of keys are desired from \code{keys} } \item{...}{other arguments. These include: \describe{ \item{pattern:}{the pattern to match (used by keys)} \item{column:}{the column to search on. This is used by keys and is for when the thing you want to pattern match is different from the keytype, or when you want to simply want to get keys that have a value for the thing specified by the column argument.} \item{fuzzy:}{TRUE or FALSE value. Use fuzzy matching? (this is used with pattern by the keys method)} } } } \value{ \code{keys},\code{columns} and \code{keytypes} each return a character vector or possible values. \code{select} returns a data.frame. } \author{Marc Carlson} \examples{ ## 1st create a Mart object and specify the dataset mart<-useMart(dataset="hsapiens_gene_ensembl",biomart='ensembl') ## you can list the keytypes keytypes(mart) ## you can list the columns columns(mart) ## And you can extract keys when this is supported for your keytype of interest k = keys(mart, keytype="chromosome_name") head(k) ## You can even do some pattern matching on the keys k = keys(mart, keytype="chromosome_name", pattern="LRG") head(k) ## Finally you can use select to extract records for things that you are ## interested in. affy=c("202763_at","209310_s_at","207500_at") select(mart, keys=affy, columns=c('affy_hg_u133_plus_2','entrezgene'), keytype='affy_hg_u133_plus_2') } \keyword{methods} biomaRt/man/useDataset.Rd0000644000175400017540000000120713175713423016347 0ustar00biocbuildbiocbuild\name{useDataset} \alias{useDataset} \title{Select a dataset to use and updates Mart object} \description{This function selects a dataset and updates the Mart object} \usage{useDataset(dataset,mart, verbose = FALSE)} \arguments{ \item{dataset}{Dataset you want to use. List of possible datasets can be retrieved using the function listDatasets} \item{mart}{Mart object created with the useMart function} \item{verbose}{Give detailed output of what the method is doing, for debugging} } \author{Steffen Durinck} \examples{ if(interactive()){ mart=useMart("ensembl") mart=useDataset("hsapiens_gene_ensembl", mart = mart) } } \keyword{methods} biomaRt/man/useEnsembl.Rd0000644000175400017540000000351213175713423016350 0ustar00biocbuildbiocbuild\name{useEnsembl} \alias{useEnsembl} \title{Connects to the selected BioMart database and dataset hosted by Ensembl} \description{A first step in using the biomaRt package is to select a BioMart database and dataset to use. The useEnsembl function enables one to connect to a specified BioMart database and dataset hosted by Ensembl without having to specify the Ensembl URL. To know which BioMart databases are available see the listEnsembl function. To know which datasets are available within a BioMart database, first select the BioMart database using useMart and then use the listDatasets function on the selected BioMart, see listDatasets function.} \usage{ useEnsembl(biomart, dataset, host="www.ensembl.org", version = NULL, GRCh = NULL, mirror = NULL, verbose = FALSE) } \arguments{ \item{biomart}{BioMart database name you want to connect to. Possible database names can be retrieved with the function \code{\link{listEnsembl}}} \item{dataset}{Dataset you want to use. To see the different datasets available within a biomaRt you can e.g. do: mart = useEnsembl('ENSEMBL_MART_ENSEMBL'), followed by listDatasets(mart).} \item{host}{Host to connect to if different then www.ensembl.org} \item{version}{Ensembl version to connect to when wanting to connect to an archived Ensembl version} \item{GRCh}{GRCh version to connect to if not the current GRCh38, currently this can only be 37} \item{mirror}{Specify an Ensembl mirror to connect to. The valid options here are 'www', 'uswest', 'useast', 'asia'. If no mirror is specified the default will be used which redirects you to the closest mirror to your location.} \item{verbose}{Give detailed output of what the method is doing while in use, for debugging} } \author{Steffen Durinck} \examples{ if(interactive()){ mart <- useEnsembl("ENSEMBL_MART_ENSEMBL") } } \keyword{methods} biomaRt/man/useMart.Rd0000644000175400017540000000512113175713423015664 0ustar00biocbuildbiocbuild\name{useMart} \alias{useMart} \title{Connects to the selected BioMart database and dataset} \description{A first step in using the biomaRt package is to select a BioMart database and dataset to use. The useMart function enables one to connect to a specified BioMart database and dataset within this database. To know which BioMart databases are available see the listMarts function. To know which datasets are available within a BioMart database, first select the BioMart database using useMart and then use the listDatasets function on the selected BioMart, see listDatasets function.} \usage{useMart(biomart, dataset, host="www.ensembl.org", path="/biomart/martservice", port=80, archive=FALSE, ssl.verifypeer = TRUE, ensemblRedirect = TRUE, version, verbose = FALSE)} \arguments{ \item{biomart}{BioMart database name you want to connect to. Possible database names can be retrieved with the functio listMarts} \item{dataset}{Dataset you want to use. To see the different datasets available within a biomaRt you can e.g. do: mart = useMart('ensembl'), followed by listDatasets(mart).} \item{host}{Host to connect to. Defaults to \code{www.ensembl.org}} \item{path}{Path that should be pasted after to host to get access to the web service URL} \item{port}{port to connect to, will be pasted between host and path} \item{archive}{Boolean to indicate if you want to access archived versions of BioMart databases. Note that this argument is now deprecated and will be removed in the future. A better alternative is to leave archive = FALSE and to specify the url of the archived BioMart you want to access. For Ensembl you can view the list of archives using \code{\link{listEnsemblArchives}}} \item{ssl.verifypeer}{Set SSL peer verification on or off. By default ssl.verifypeer is set to TRUE} \item{ensemblRedirect}{By default when you access Ensembl BioMart it will redirect you to your local mirror, even if you have set a region specific mirror in the \code{host} argument. By setting this argument to \code{FALSE} you can override this behaviour and force access to your specified \code{host}. Defaults to \code{TRUE} to prevent overwhelming the main Ensembl site. If you are accessing a BioMart instance other than Ensembl this should have no effect.} \item{version}{Use version name instead of biomart name to specify which BioMart you want to use} \item{verbose}{Give detailed output of what the method is doing while in use, for debugging} } \author{Steffen Durinck} \examples{ if(interactive()){ mart = useMart("ensembl") mart=useMart(biomart="ensembl", dataset="hsapiens_gene_ensembl") } } \keyword{methods} biomaRt/tests/0000755000175400017540000000000013175713423014345 5ustar00biocbuildbiocbuildbiomaRt/tests/testthat/0000755000175400017540000000000013175713423016205 5ustar00biocbuildbiocbuildbiomaRt/tests/testthat.R0000644000175400017540000000007113175713423016326 0ustar00biocbuildbiocbuildlibrary(testthat) library(biomaRt) test_check("biomaRt")biomaRt/tests/testthat/test_colnames.R0000644000175400017540000000327513175713423021177 0ustar00biocbuildbiocbuildlibrary(biomaRt) ensembl=useMart("ensembl") ensembl = useDataset("hsapiens_gene_ensembl", mart=ensembl) context('Testing column name assignments generation') ## create two test data.frames bad_result <- data.frame("Not a real column" = 1:2, "Ensembl Gene ID" = 3:4, check.names = FALSE) good_result <- data.frame("Chromosome/scaffold name" = 1:2, "Gene stable ID" = 3:4, check.names = FALSE) ## this should warn that we can't match one of the column names (hopefully this never happens for real) expect_warning(.setResultColNames(result = bad_result, mart = ensembl, attributes = c('chromosome_name', 'ensembl_gene_id')), "Problems assigning column names") ## check the reassignment of colnames from 'description' to 'name' expect_equal(colnames(.setResultColNames(result = good_result, mart = ensembl, attributes = c('chromosome_name', 'ensembl_gene_id'))), c("chromosome_name", "ensembl_gene_id")) ## check we reorder them if needed expect_equal(colnames(.setResultColNames(result = good_result, mart = ensembl, attributes = c('ensembl_gene_id', 'chromosome_name'))), c("ensembl_gene_id", "chromosome_name")) ## check we can handle the ambiguous description field in some datasets attributes=c("ensembl_transcript_id", "cdna", ## Description is 'Query protein or transcript ID' lots of matches "neugenii_homolog_canonical_transcript_protein") mart <- useMart(biomart = "ensembl", host = "www.ensembl.org", dataset ="mmusculus_gene_ensembl") seq <- getBM(filter = "ensembl_gene_id", values = "ENSMUSG00000000103", attributes = attributes, mart = mart) expect_equal(colnames(seq), attributes) biomaRt/tests/testthat/test_hostProcessing.R0000644000175400017540000000074613175713423022410 0ustar00biocbuildbiocbuildlibrary(biomaRt) ## adding http if needed host <- 'www.myurl.org' expect_equal(object = .cleanHostURL(host = host), expected = "http://www.myurl.org") ## stripping trailing slash host <- 'http://www.myurl.org/' expect_equal(object = .cleanHostURL(host = host), expected = "http://www.myurl.org") ## leave https already there host <- 'https://www.myurl.org' expect_equal(object = .cleanHostURL(host = host), expected = "https://www.myurl.org") biomaRt/tests/testthat/test_useMart.R0000644000175400017540000000073413175713423021013 0ustar00biocbuildbiocbuildlibrary(biomaRt) ## checking the show() method ensembl <- useMart("ensembl") ensembl_with_dataset <- useDataset(ensembl, dataset = "xtropicalis_gene_ensembl") test_that("Show give sensible dataset information", { expect_output(object = show(ensembl), regexp = "No dataset selected") expect_output(object = show(ensembl_with_dataset), regexp = "Using the xtropicalis_gene_ensembl dataset") }) biomaRt/vignettes/0000755000175400017540000000000013214603701015202 5ustar00biocbuildbiocbuildbiomaRt/vignettes/biomaRt.Rmd0000644000175400017540000006544713175713423017274 0ustar00biocbuildbiocbuild--- title: "The biomaRt users guide" author: "Steffen Durinck, Wolfgang Huber, Mike Smith" package: "`r pkg_ver('biomaRt')`" output: BiocStyle::html_document: md_extensions: "-autolink_bare_uris" vignette: > %\VignetteIndexEntry{The biomaRt users guide} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- ```{r setup, cache = F, echo = FALSE} knitr::opts_chunk$set(error = TRUE) ``` # Introduction In recent years a wealth of biological data has become available in public data repositories. Easy access to these valuable data resources and firm integration with data analysis is needed for comprehensive bioinformatics data analysis. The `r Biocpkg("biomaRt")` package, provides an interface to a growing collection of databases implementing the [BioMart software suite](http://www.biomart.org). The package enables retrieval of large amounts of data in a uniform way without the need to know the underlying database schemas or write complex SQL queries. Examples of BioMart databases are Ensembl, Uniprot and HapMap. These major databases give `r Biocpkg("biomaRt")` users direct access to a diverse set of data and enable a wide range of powerful online queries from R. # Selecting a BioMart database and dataset Every analysis with `r Biocpkg("biomaRt")` starts with selecting a BioMart database to use. A first step is to check which BioMart web services are available. The function `listMarts()` will display all available BioMart web services ```{r annotate,echo=FALSE} ## library("annotate") options(width=120) ``` ```{r biomaRt} library("biomaRt") listMarts() ``` Note: if the function `useMart()` runs into proxy problems you should set your proxy first before calling any `r Biocpkg("biomaRt")` functions. You can do this using the Sys.putenv command: ```{r putenv, eval = FALSE} Sys.setenv("http_proxy" = "http://my.proxy.org:9999") ``` Some users have reported that the workaround above does not work, in this case an alternative proxy solution below can be tried: ```{r rCurlOptions, eval = FALSE} options(RCurlOptions = list(proxy="uscache.kcc.com:80",proxyuserpwd="------:-------")) ``` The `useMart()` function can now be used to connect to a specified BioMart database, this must be a valid name given by `listMarts()`. In the next example we choose to query the Ensembl BioMart database. ```{r ensembl1} ensembl=useMart("ensembl") ``` BioMart databases can contain several datasets, for Ensembl every species is a different dataset. In a next step we look at which datasets are available in the selected BioMart by using the function `listDatasets()`. ```{r listDatasets} listDatasets(ensembl) ``` To select a dataset we can update the `Mart` object using the function `useDataset()`. In the example below we choose to use the hsapiens dataset. ```{r ensembl2, eval=TRUE} ensembl = useDataset("hsapiens_gene_ensembl",mart=ensembl) ``` Or alternatively if the dataset one wants to use is known in advance, we can select a BioMart database and dataset in one step by: ```{r ensembl3} ensembl = useMart("ensembl",dataset="hsapiens_gene_ensembl") ``` # How to build a biomaRt query The `getBM()` function has three arguments that need to be introduced: filters, attributes and values. *Filters* define a restriction on the query. For example you want to restrict the output to all genes located on the human X chromosome then the filter *chromosome_name* can be used with value 'X'. The `listFilters()` function shows you all available filters in the selected dataset. ```{r filters} filters = listFilters(ensembl) filters[1:5,] ``` *Attributes* define the values we are interested in to retrieve. For example we want to retrieve the gene symbols or chromosomal coordinates. The `listAttributes()` function displays all available attributes in the selected dataset. ```{r attributes} attributes = listAttributes(ensembl) attributes[1:5,] ``` The `getBM()` function is the main query function in `r Biocpkg("biomaRt")`. It has four main arguments: * `attributes`: is a vector of attributes that one wants to retrieve (= the output of the query). * `filters`: is a vector of filters that one wil use as input to the query. * `values`: a vector of values for the filters. In case multple filters are in use, the values argument requires a list of values where each position in the list corresponds to the position of the filters in the filters argument (see examples below). * `mart`: is an object of class `Mart`, which is created by the `useMart()` function. *Note: for some frequently used queries to Ensembl, wrapper functions are available: `getGene()` and `getSequence()`. These functions call the `getBM()` function with hard coded filter and attribute names.* Now that we selected a BioMart database and dataset, and know about attributes, filters, and the values for filters; we can build a `r Biocpkg("biomaRt")` query. Let's make an easy query for the following problem: We have a list of Affymetrix identifiers from the u133plus2 platform and we want to retrieve the corresponding EntrezGene identifiers using the Ensembl mappings. The u133plus2 platform will be the filter for this query and as values for this filter we use our list of Affymetrix identifiers. As output (attributes) for the query we want to retrieve the EntrezGene and u133plus2 identifiers so we get a mapping of these two identifiers as a result. The exact names that we will have to use to specify the attributes and filters can be retrieved with the `listAttributes()` and `listFilters()` function respectively. Let's now run the query: ```{r getBM1, echo=TRUE,eval=TRUE} affyids=c("202763_at","209310_s_at","207500_at") getBM(attributes=c('affy_hg_u133_plus_2', 'entrezgene'), filters = 'affy_hg_u133_plus_2', values = affyids, mart = ensembl) ``` # Examples of biomaRt queries In the sections below a variety of example queries are described. Every example is written as a task, and we have to come up with a `r Biocpkg("biomaRt")` solution to the problem. ## Annotate a set of Affymetrix identifiers with HUGO symbol and chromosomal locations of corresponding genes We have a list of Affymetrix hgu133plus2 identifiers and we would like to retrieve the HUGO gene symbols, chromosome names, start and end positions and the bands of the corresponding genes. The `listAttributes()` and the `listFilters()` functions give us an overview of the available attributes and filters and we look in those lists to find the corresponding attribute and filter names we need. For this query we'll need the following attributes: hgnc_symbol, chromsome_name, start_position, end_position, band and affy_hg_u133_plus_2 (as we want these in the output to provide a mapping with our original Affymetrix input identifiers. There is one filter in this query which is the affy_hg_u133_plus_2 filter as we use a list of Affymetrix identifiers as input. Putting this all together in the `getBM()` and performing the query gives: ```{r task1, echo=TRUE,eval=TRUE} affyids=c("202763_at","209310_s_at","207500_at") getBM(attributes = c('affy_hg_u133_plus_2', 'hgnc_symbol', 'chromosome_name', 'start_position', 'end_position', 'band'), filters = 'affy_hg_u133_plus_2', values = affyids, mart = ensembl) ``` ## Annotate a set of EntrezGene identifiers with GO annotation In this task we start out with a list of EntrezGene identiers and we want to retrieve GO identifiers related to biological processes that are associated with these entrezgene identifiers. Again we look at the output of `listAttributes()` and `listFilters()` to find the filter and attributes we need. Then we construct the following query: ```{r task2, echo=TRUE,eval=TRUE} entrez=c("673","837") goids = getBM(attributes = c('entrezgene', 'go_id'), filters = 'entrezgene', values = entrez, mart = ensembl) head(goids) ``` ## Retrieve all HUGO gene symbols of genes that are located on chromosomes 17,20 or Y, and are associated with specific GO terms The GO terms we are interested in are: **GO:0051330**, **GO:0000080**, **GO:0000114**, **GO:0000082**. The key to performing this query is to understand that the `getBM()` function enables you to use more than one filter at the same time. In order to do this, the filter argument should be a vector with the filter names. The values should be a list, where the first element of the list corresponds to the first filter and the second list element to the second filter and so on. The elements of this list are vectors containing the possible values for the corresponding filters. ```{r task3, echo=TRUE,eval=TRUE} go=c("GO:0051330","GO:0000080","GO:0000114","GO:0000082") chrom=c(17,20,"Y") getBM(attributes= "hgnc_symbol", filters=c("go_id","chromosome_name"), values=list(go, chrom), mart=ensembl) ``` ## Annotate set of idenfiers with INTERPRO protein domain identifiers In this example we want to annotate the following two RefSeq identifiers: **NM_005359** and **NM_000546** with INTERPRO protein domain identifiers and a description of the protein domains. ```{r task4, echo=TRUE,eval=TRUE} refseqids = c("NM_005359","NM_000546") ipro = getBM(attributes=c("refseq_mrna","interpro","interpro_description"), filters="refseq_mrna", values=refseqids, mart=ensembl) ipro ``` ## Select all Affymetrix identifiers on the hgu133plus2 chip and Ensembl gene identifiers for genes located on chromosome 16 between basepair 1100000 and 1250000. In this example we will again use multiple filters: *chromosome_name*, *start*, and *end* as we filter on these three conditions. Note that when a chromosome name, a start position and an end position are jointly used as filters, the BioMart webservice interprets this as return everything from the given chromosome between the given start and end positions. ```{r task5, eval = TRUE} getBM(attributes = c('affy_hg_u133_plus_2','ensembl_gene_id'), filters = c('chromosome_name','start','end'), values = list(16,1100000,1250000), mart = ensembl) ``` ## Retrieve all entrezgene identifiers and HUGO gene symbols of genes which have a "MAP kinase activity" GO term associated with it. The GO identifier for MAP kinase activity is **GO:0004707**. In our query we will use *go_id* as our filter, and *entrezgene* and *hgnc_symbol* as attributes. Here's the query: ```{r task6, echo=TRUE, eval = TRUE} getBM(attributes = c('entrezgene','hgnc_symbol'), filters = 'go', values = 'GO:0004707', mart = ensembl) ``` ## Given a set of EntrezGene identifiers, retrieve 100bp upstream promoter sequences All sequence related queries to Ensembl are available through the `getSequence()` wrapper function. `getBM()` can also be used directly to retrieve sequences but this can get complicated so using getSequence is recommended. Sequences can be retrieved using the `getSequence()` function either starting from chromosomal coordinates or identifiers. The chromosome name can be specified using the *chromosome* argument. The *start* and *end* arguments are used to specify *start* and *end* positions on the chromosome. The type of sequence returned can be specified by the *seqType* argument which takes the following values: * *cdna* * *peptide* for protein sequences * *3utr* for 3' UTR sequences * *5utr* for 5' UTR sequences * *gene_exon* for exon sequences only * *transcript_exon* for transcript specific exonic sequences only * *transcript_exon_intron* gives the full unspliced transcript, that is exons + introns * *gene_exon_intron* gives the exons + introns of a gene * *coding* gives the coding sequence only * *coding_transcript_flank* gives the flanking region of the transcript including the UTRs, this must be accompanied with a given value for the upstream or downstream attribute * *coding_gene_flank* gives the flanking region of the gene including the UTRs, this must be accompanied with a given value for the upstream or downstream attribute * *transcript_flank* gives the flanking region of the transcript exculding the UTRs, this must be accompanied with a given value for the upstream or downstream attribute * *gene_flank* gives the flanking region of the gene excluding the UTRs, this must be accompanied with a given value for the upstream or downstream attribute In MySQL mode the `getSequence()` function is more limited and the sequence that is returned is the 5' to 3'+ strand of the genomic sequence, given a chromosome, as start and an end position. This task requires us to retrieve 100bp upstream promoter sequences from a set of EntrzGene identifiers. The *type* argument in `getSequence()` can be thought of as the filter in this query and uses the same input names given by `listFilters()`. In our query we use entrezgene for the type argument. Next we have to specify which type of sequences we want to retrieve, here we are interested in the sequences of the promoter region, starting right next to the coding start of the gene. Setting the *seqType* to coding_gene_flank will give us what we need. The *upstream* argument is used to specify how many bp of upstream sequence we want to retrieve, here we'll retrieve a rather short sequence of 100bp. Putting this all together in `getSequence()` gives: ```{r task7, eval=TRUE} entrez=c("673","7157","837") getSequence(id = entrez, type="entrezgene", seqType="coding_gene_flank", upstream=100, mart=ensembl) ``` ## Retrieve all 5' UTR sequences of all genes that are located on chromosome 3 between the positions 185,514,033 and 185,535,839 As described in the provious task getSequence can also use chromosomal coordinates to retrieve sequences of all genes that lie in the given region. We also have to specify which type of identifier we want to retrieve together with the sequences, here we choose for entrezgene identifiers. ```{r task8, echo=TRUE,eval=TRUE} utr5 = getSequence(chromosome=3, start=185514033, end=185535839, type="entrezgene", seqType="5utr", mart=ensembl) utr5 ``` ## Retrieve protein sequences for a given list of EntrezGene identifiers In this task the type argument specifies which type of identifiers we are using. To get an overview of other valid identifier types we refer to the `listFilters()` function. ```{r task9, echo=TRUE, eval=TRUE} protein = getSequence(id=c(100, 5728), type="entrezgene", seqType="peptide", mart=ensembl) protein ``` ## Retrieve known SNPs located on the human chromosome 8 between positions 148350 and 148612 For this example we'll first have to connect to a different BioMart database, namely snp. ```{r task10, echo=TRUE, eval=TRUE} snpmart = useMart(biomart = "ENSEMBL_MART_SNP", dataset="hsapiens_snp") ``` The `listAttributes()` and `listFilters()` functions give us an overview of the available attributes and filters. From these we need: *refsnp_id*, *allele*, *chrom_start* and *chrom_strand* as attributes; and as filters we'll use: *chrom_start*, *chrom_end* and *chr_name*. Note that when a chromosome name, a start position and an end position are jointly used as filters, the BioMart webservice interprets this as return everything from the given chromosome between the given start and end positions. Putting our selected attributes and filters into getBM gives: ```{r task10b} getBM(attributes = c('refsnp_id','allele','chrom_start','chrom_strand'), filters = c('chr_name','start','end'), values = list(8,148350,148612), mart = snpmart) ``` ## Given the human gene TP53, retrieve the human chromosomal location of this gene and also retrieve the chromosomal location and RefSeq id of its homolog in mouse. The `getLDS()` (Get Linked Dataset) function provides functionality to link 2 BioMart datasets which each other and construct a query over the two datasets. In Ensembl, linking two datasets translates to retrieving homology data across species. The usage of getLDS is very similar to `getBM()`. The linked dataset is provided by a separate `Mart` object and one has to specify filters and attributes for the linked dataset. Filters can either be applied to both datasets or to one of the datasets. Use the listFilters and listAttributes functions on both `Mart` objects to find the filters and attributes for each dataset (species in Ensembl). The attributes and filters of the linked dataset can be specified with the attributesL and filtersL arguments. Entering all this information into `getLDS()` gives: ```{r getLDS} human = useMart("ensembl", dataset = "hsapiens_gene_ensembl") mouse = useMart("ensembl", dataset = "mmusculus_gene_ensembl") getLDS(attributes = c("hgnc_symbol","chromosome_name", "start_position"), filters = "hgnc_symbol", values = "TP53",mart = human, attributesL = c("refseq_mrna","chromosome_name","start_position"), martL = mouse) ``` # Using archived versions of Ensembl It is possible to query archived versions of Ensembl through `r Biocpkg("biomaRt")`. `r Biocpkg("biomaRt")` provides the function `listEnsemblArchives()` to view the available archives. This function takes no arguments, and produces a table containing the names of the available archived versions, the date they were first available, and the URL where they can be accessed. ```{r archiveMarts, echo = TRUE, eval = TRUE} listEnsemblArchives() ``` Alternatively, one can use the website to find archived version. From the main page scroll down the bottom of the page, click on 'view in Archive' and select the archive you need. *You will notice that there is an archive URL even for the current release of Ensembl. It can be useful to use this if you wish to ensure that script you write now will return exactly the same results in the future. Using `www.ensembl.org` will always access the current release, and so the data retrieved may change over time as new releases come out.* Whichever method you use to find the URL of the archive you wish to query, copy the url and use that in the `host` argument as shown below to connect to the specified BioMart database. The example below shows how to query Ensembl 54. ```{r archiveMarts3, echo = TRUE, eval = TRUE} listMarts(host = 'may2009.archive.ensembl.org') ensembl54 <- useMart(host='may2009.archive.ensembl.org', biomart='ENSEMBL_MART_ENSEMBL', dataset='hsapiens_gene_ensembl') ``` # Using a BioMart other than Ensembl To demonstrate the use of the `r Biocpkg("biomaRt")` package with non-Ensembl databases the next query is performed using the Wormbase ParaSite BioMart. In this example, we use the `listMarts()` function to find the name of the available marts, given the URL of Wormbase. We use this to connect to Wormbase BioMart, find and select the gene dataset, and print the first 6 available attributes and filters. Then we use a list of gene names as filter and retrieve associated transcript IDs and the transcript biotype. ```{r wormbase, echo=TRUE, eval=TRUE} listMarts(host = "parasite.wormbase.org") wormbase = useMart(biomart = "parasite_mart", host = "parasite.wormbase.org") listDatasets(wormbase) wormbase <- useDataset(mart = wormbase, dataset = "wbps_gene") head(listFilters(wormbase)) head(listAttributes(wormbase)) getBM(attributes = c("external_gene_id", "wbps_transcript_id", "transcript_biotype"), filters="gene_name", values=c("unc-26","his-33"), mart=wormbase) ``` # biomaRt helper functions This section describes a set of `r Biocpkg("biomaRt")` helper functions that can be used to export FASTA format sequences, retrieve values for certain filters and exploring the available filters and attributes in a more systematic manner. ## exportFASTA The data.frames obtained by the getSequence function can be exported to FASTA files using the `exportFASTA()` function. One has to specify the data.frame to export and the filename using the file argument. ## Finding out more information on filters ### filterType Boolean filters need a value TRUE or FALSE in `r Biocpkg("biomaRt")`. Setting the value TRUE will include all information that fulfill the filter requirement. Setting FALSE will exclude the information that fulfills the filter requirement and will return all values that don't fulfill the filter. For most of the filters, their name indicates if the type is a boolean or not and they will usually start with "with". However this is not a rule and to make sure you got the type right you can use the function `filterType()` to investigate the type of the filter you want to use. ```{r filterType} filterType("with_affy_hg_u133_plus_2",ensembl) ``` ### filterOptions Some filters have a limited set of values that can be given to them. To know which values these are one can use the `filterOptions()` function to retrieve the predetermed values of the respective filter. ```{r filterOptions} filterOptions("biotype",ensembl) ``` If there are no predetermed values e.g. for the entrezgene filter, then `filterOptions()` will return the type of filter it is. And most of the times the filter name or it's description will suggest what values one case use for the respective filter (e.g. entrezgene filter will work with enterzgene identifiers as values) ## Attribute Pages For large BioMart databases such as Ensembl, the number of attributes displayed by the `listAttributes()` function can be very large. In BioMart databases, attributes are put together in pages, such as sequences, features, homologs for Ensembl. An overview of the attributes pages present in the respective BioMart dataset can be obtained with the `attributePages()` function. ```{r attributePages} pages = attributePages(ensembl) pages ``` To show us a smaller list of attributes which belong to a specific page, we can now specify this in the `listAttributes()` function. *The set of attributes is still quite long, so we use `head()` to show only the first few items here.* ```{r listAttributes} head(listAttributes(ensembl, page="feature_page")) ``` We now get a short list of attributes related to the region where the genes are located. # Local BioMart databases The `r Biocpkg("biomaRt")` package can be used with a local install of a public BioMart database or a locally developed BioMart database and web service. In order for `r Biocpkg("biomaRt")` to recognize the database as a BioMart, make sure that the local database you create has a name conform with ` database_mart_version ` where database is the name of the database and version is a version number. No more underscores than the ones showed should be present in this name. A possible name is for example ` ensemblLocal_mart_46 `. ## Minimum requirements for local database installation More information on installing a local copy of a BioMart database or develop your own BioMart database and webservice can be found on Once the local database is installed you can use `r Biocpkg("biomaRt")` on this database by: ```{r localCopy, eval = FALSE} listMarts(host="www.myLocalHost.org", path="/myPathToWebservice/martservice") mart=useMart("nameOfMyMart",dataset="nameOfMyDataset",host="www.myLocalHost.org", path="/myPathToWebservice/martservice") ``` For more information on how to install a public BioMart database see: http://www.biomart.org/install.html and follow link databases. # Using `select()` In order to provide a more consistent interface to all annotations in Bioconductor the `select()`, `columns()`, `keytypes()` and `keys()` have been implemented to wrap some of the existing functionality above. These methods can be called in the same manner that they are used in other parts of the project except that instead of taking a `AnnotationDb` derived class they take instead a `Mart` derived class as their 1st argument. Otherwise usage should be essentially the same. You still use `columns()` to discover things that can be extracted from a `Mart`, and `keytypes()` to discover which things can be used as keys with `select()`. ```{r columnsAndKeyTypes} mart <- useMart(dataset="hsapiens_gene_ensembl",biomart='ensembl') head(keytypes(mart), n=3) head(columns(mart), n=3) ``` And you still can use `keys()` to extract potential keys, for a particular key type. ```{r keys1} k = keys(mart, keytype="chromosome_name") head(k, n=3) ``` When using `keys()`, you can even take advantage of the extra arguments that are available for others keys methods. ```{r keys2} k = keys(mart, keytype="chromosome_name", pattern="LRG") head(k, n=3) ``` Unfortunately the `keys()` method will not work with all key types because they are not all supported. But you can still use `select()` here to extract columns of data that match a particular set of keys (this is basically a wrapper for `getBM()`). ```{r select} affy=c("202763_at","209310_s_at","207500_at") select(mart, keys=affy, columns=c('affy_hg_u133_plus_2','entrezgene'), keytype='affy_hg_u133_plus_2') ``` So why would we want to do this when we already have functions like `getBM()`? For two reasons: 1) for people who are familiar with select and it's helper methods, they can now proceed to use `r Biocpkg("biomaRt")` making the same kinds of calls that are already familiar to them and 2) because the select method is implemented in many places elsewhere, the fact that these methods are shared allows for more convenient programmatic access of all these resources. An example of a package that takes advantage of this is the `r Biocpkg("OrganismDbi")` package. Where several packages can be accessed as if they were one resource. # Session Info ```{r sessionInfo} sessionInfo() warnings() ```